Timeline: History of Deep Learning

2014 BCE

Skype has Real Time Translation

2012 BCE

"Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared view of four research groups"

The research groups of Microsoft, Google, IBM and Hinton´s lab shows results of working on Neural Nets

2012 BCE

"ImageNet Classification with deep convolutional neural networks"

By Hinton, Krizhevsky and Sutskever. They create an entry to the ILSVRC (Large Scale Visual Recognition Competition). It was the climax of deep learning ascent.

2011 BCE

Advances on Google's and Android's speech recognition system

Navdeep Jaitly works on Google's speech recognition system. It pushed Android's speech recognition algorithm

2011 BCE

Google Brain is created.

Andrew Ng and Jeff Dean create Google Brain to make experiments with neural nets using a great number of CPU cores.

2009 BCE

"Large Scale Deep Unsupervised Learning using Graphic Processors"

By Raina, Madhavan, Ng. It suggest taht unsupervised learning on speech recognition is 70 times faster using GPUs.

2007 BCE

"Greedy layer-wise Training of Deep Networks"

By Bengio et al. It has arguments to say that deep machine learning methods are more efficient for dificult problems than shallow methods

2006 BCE

"A Fast Learning Algorithm for Deep Belief Nets"

By Hinton, Osindero, Whye. A breakthrough significant enough to rekindle interest in neural nets.

Period: 2006 BCE to 2017 BCE

Deep Learning move have born.

2002 BCE

"Training products of experts by minimizing contrastive divergence"

By Hinton. It showed that Restricted Boltzmann machine can be trained in an efficient manner.

1997 BCE

New concept of "Long Short Term Memory" LSTM

Introduced by Schmidhuber and Hochreifer

1995 BCE

The Helmholtz Machine By Hinton, Dayan, Frey and Neal

This type of machine was born in "The Wake-sleep algorithm unsupervised neural networks"

1995 BCE

"Learning to play the game of chess"

By Sebatian Thrun. It was a demostration of problems of TD-Gammon (reinforcement learning) approach

1995 BCE

"Convolutional Networks for Images, Speech, and Time-Series"

By the modern giant of deep learning Yoshua Bengio

Period: 1995 BCE to 2002 BCE

Second AI Winter

AI Winter began when "Support Vector Machines" appear.

1993 BCE

Reinforcement Learning

Was treated in the PhD thesis "" Reinforcement learning for robots using neural networks"

1993 BCE

"A connectionist Approach to Speech Recognition"

By Bengio. It explains the general failure of Recurrent Neural Nets (RNN)

1992 BCE

Belief Nets appears

Thank to Redford M. Neal in "Connectionist learning of belief networks". Theese nets are like Boltzman Machines but with layers

1990 BCE

Boom of CNN in handwritten zip code recognition

LeCun's CNN system is used on 10 to 20% of all the checks in U.S

1989 BCE

"Multilayer feedforward networks are universal approximators"

It Mathematically proved that multilayers allow neural nets to theoretically implement any function

1989 BCE

"Backpropagation applied to handwritten zip code recognition"

Yann LeCun et al. at AT&T Bell Labs.

1989 BCE

Early Neural Net applications on Robotics

In CMU Navlab was created "ALVINN: An autonomous land vehicle in a neural network"

1989 BCE

"Phoneme Recognition using Time Delay Neural Networks"

By Waibel, Hanazawa, Hinton, Shikano, Lang. Speech Recognition close up begins with this article.

1987 BCE

CIFAR funded Hinton's work

1986 BCE

Backpropagation Neural Nets return to Popularity

1986 BCE

"Learning Representations by backpropagation errors"

David Rumelhart, Geoffrey Hinton and Ronald Williams publish this paper that talks about the problems discussed about Perceptrons by Minsky

1986 BCE

"Weight Sharing" or convolutional neural nets.

It was discussed in the analisys of backpropagation by Rumelhart, Hinton and Williams

1986 BCE

Autoencoders by Hinton, Rumelhart and Williams

The idea of Autoencoders is discussed in the analysis of backpropagation

1985 BCE

"A Learning Algorithm for Boltzmann Machine" by Ackley, Hinton, Zejnowski

Boltzmann Machines are networks just like neural nets and have units that are very similar to Perceptrons, theese units are stochastic, it means, they behave according to a probability distribution.

1982 BCE