Geoffrey Hinton's Google Tech Talk linking machine learning to biology, explaining dropout as a form of model averaging and why sexual reproduction and stochastic spiking neurons may prevent overfitting.
Why Hacker News points people here: Recommended as a simple, elegant explanation of dropout and its connection to how the brain works.
“If you want a great overview of the dropout technique, watch this tech talk by Hinton”HN comment, 2013
Hinton explains dropout and its link to how brains work.
Full comment
Here is a GREAT talk by Geoffrey Hinton (the Prof running said lab) this video where he explains the method. Unfortunately, even though it was posted three times to HN link it never made the fron page. Here is the my summary and comment: " Great talk. I don't know much about artificial neural networks (ANN) and even less about natural ones, but I have the feeling that I learnt a lot from this video. If I understand correct, Hinton uses so many artificial neurons compared to the amount of learning data, that you would usually see an overfitting effect. However, his ANN's randomly shut of a substantial part (~50%) of the neurons during each learning iteration. He calls this "dropout". Therefore, a single ANN represents many different models. Most models never get trained, but they exist in the ANN, because they share their weights with the trained models. This learning method avoids over specializing and therefore improves robustness with respect to new data but it also allows for arbitrary combination of different models which tremendously enlarges the pool of testable models. When using or testing these ANNs you also "dropout" neurons during every prediction. Practically, every rerun predicts a different result by using a different model. Afterwards, these results are averaged. The more results, the higher the chance, that the classification is correct. Hinton argues, that our brains work in a similar way. This explains among other things a) Why are neurons firing in a random
Strongly recommendspiloochMay 2013
“the extraordinary simple and elegant talk by G. Hinton last summer”
Simple, elegant talk linking dropout to why neurons use spikes.
Full comment
It seems the relationship between the brain and deep learning has evolved in such a way that the later can help with insights into how the former works. In this regard, I thought I would mention the extraordinary simple and elegant talk by G. Hinton last summer: this video It starts from a simple and clever improvement to an existing deep learning method and ends up with beautiful (and simple!) insights on why neurons are using simple spikes to communicate.
Strongly recommendseveningcoffeeNov 2015
“I would highly recommend this video”
Hinton's 2012 talk related to low-multiplication neural network research.
Full comment
I would highly recommend this video this video from Geoffrey Hinton in 2012 that I think is related to this research.
Recommendspsb217Dec 2012
“watch Hinton describe it in a recent talk he gave at Google”
Hinton describes dropout training as model averaging.
Full comment
In addition to what bravura said, Geoff Hinton's group at UT has recently introduced a new approach to training neural networks that they call "dropout". You can read about it in their paper ( link ) or watch Hinton describe it in a recent talk he gave at Google ( this video ). Roughly speaking, dropout training provides a strong regularizing effect through a sort of model averaging that is conceptually related to the well-known bagging approach from which random forests derive their power and flexibility. Dropout training has already produced state-of-the art results on several time-worn standard benchmarks and helped Hinton's group win a recent Kaggle competition (for an overview of their approach, see: link ). I've played around with this a bit over the last few weeks, and have a Matlab implementation publicly available from my Github at: link .
RecommendslightcatcherMay 2013
“If you want a great overview of the dropout technique, watch this tech talk by Hinton”
Great overview of the dropout technique.
Full comment
Using dropout with deep neural networks is an (extremely cheap) way to gain many of the benefits of using an ensemble. If you want a great overview of the dropout technique, watch this tech talk by Hinton: this video Also, the recent maxout algorithm (from Montreal group, authors of Theano) that got state of the art results on several datasets is essentially just an algorithm designed to do particularly well with dropout (as far as I understand it).
RecommendstaliesinbMay 2013
“has a very stimulating pair of hypotheses related to this”
Hinton on dropout, bagging, and why sex and spiking neurons might prevent overfitting.
Full comment
Speaking of bagging: Geoff Hinton talks a bit about model averaging and has a very stimulating pair of hypotheses related to this, which are: a) genetic recombination in the form of sex might be "bagging across genes" that prevents tight co-adaptation (= overfitting on the evolutionary timescale) b) the brain uses noisy discrete firing instead of continuous communication because it allows for "bagging across network topologies". The bulk of his talk is about how dropout in neural nets can be used to accomplish some of the same performance benefits that model averaging gives on other algorithms (at a much smaller relative computational cost). Here's the talk: this video
RecommendscarlobFeb 2014
“sheds interesting light on some variants of deep learning and the way the brain uses (classical) noise”
Sheds light on deep learning variants and the brain's use of noise.
Full comment
As far as I know there is a decent amount of information against the fact that brains are quantum computers. On the other hand there is a talk by Hinton on youtube [1] that sheds interesting light on some variants of deep learning and the way the brain uses (classical) noise. [1] this video
Cites itiandanforthOct 2012
“He gave a Google Tech Talk on this back in June.”
Hinton's tech talk on dropout.
Full comment
Hinton's team (SuperVision) uses an interesting 'dropout' technique. He gave a Google Tech Talk on this back in June. this video And an older talk that covers some of what a deep convolutional net is: link
Cites itcs702Mar 2013
“For an overview of deep belief networks, see these videos”
One of several videos giving an overview of deep belief networks.
Full comment
Something really interesting must be happening with AI at Google, because in the past few months both Ray Kurzweil (the best-known proponent of the singularity) and Geoff Hinton (the crazy-talented individual who invented deep-belief networks using interconnected "restricted Boltzman machines"[1]) have joined the company. -- [1] For an overview of deep belief networks, see these videos: link , link , and this video
Cites itthisisdaveFeb 2014
Cited as possible source for one-bit neuron information claim.
Full comment
>As for contrastive divergence with Restricted Boltzmann Machines overcoming the vanishing gradient problem. That's true, but has anyone even tried doing this with 7-bit floating point and demonstrated it even works? I wasn't referring to RBMs here, although it does look like I misremembered the details. See slides 74-76 of this presentation for a brief sketch [0]. In these feed-forward MLPs, the information content from each neuron is capped at one bit (which is quite a bit less than seven). I seem to recall him saying something similar (and with more detail) in a Google Tech Talk a few months later [1], although I'm not sure. [0]: link [1]: this video
Cites itHoushalterNov 2015
“Geoff Hinton has a theory that Naive Bayes is just logistic regression with dropout”
Hinton's theory relating naive Bayes to logistic regression with dropout.
Full comment
Geoff Hinton has a theory that Naive Bayes is just logistic regression with dropout: this video And that if you divide all the parameters by a certain amount, you would get accurate probability estimates.
Cites ittcJun 2017
“Geoffrey Hinton described doing this back in 2012.”
Hinton described stochastic binary training, generalization, and biological motivation in 2012.
Full comment
Geoffrey Hinton described doing this back in 2012. He noted that it learned more slowly but generalized better, he described how this is nearly isomorphic with dropout, and he presented a biological motivation: this videothis video
