Hacker News Reader

"Anything humans can do in 0.1 sec, the right big 10-layer network can do too"

Actually this was argued by Connectionists in 1980s. It is called Feldman's 100-step rule:

The critical resource that is most obvious is time. Neurons whose basic computational speed is a few milliseconds must be made to account for complex behaviors which are carried out in a few hundred milliseconds (Posner, 1978). This means that entire complex behaviors are carried out in less than a hundred time steps. Current AI and simulation programs require millions of time steps.

Feldman, J. A., & Ballard, D. H. (1982). Connectionist models and their properties. Cognitive Science, 6, p. 206.

"Anything humans can do in 0.1 sec, the right big 10-layer network can do too"

That's an interesting direction from which to view things.

Then AI progress can be measured by increasing that timeframe.

Though I suspect there are some pretty gigantic discontinuities in there. The things humans can do in 3-4 seconds are qualitatively different from what they can do in less than 1 second, for example.

Still, it's a useful perspective to keep in mind.

There's also the issue that we don't even fully understand what the brain even does, so how can we claim that anything in this incomplete set of operations is possible with a computer...

> Page 60-70 talks about paragraph embedding. I haven't seen this published before.

Paragraph embedding was published this year as "Distributed Representations of Sentences and Documents". http://arxiv.org/abs/1405.4053

Thanks.

It's interesting how the state of the art is outpacing publishing.

From a quick scan that appears quite similar to the approach in papers like "Parsing Natural Scenes and Natural Language with Recursive Neural Networks" (2011)[1]. Edit: I see they cite this paper too.

[1] http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf

There's also better embeddings than word2vec now:

http://nlp.stanford.edu/projects/glove/

The characterisation of Glove as better than Word2Vec is controversial. I'm on mobile now, but one of the word2vec authors had a Google doc going through the claims, and pointing out that similar performance was possible from word2vec by changing the parameters word2vec is used with.

Edit: a link about this. https://news.ycombinator.com/item?id=8660624

Speaking from personal experience. I get paid to do deep learning. One of skymind's biggest app areas is text.

That being said: I will be benchmarking deeplearning4j's glove with word2vec here soon. Any machine learning algorithm is better when you tune it.

I personally like glove due to having less knobs. The mechanics involving document statistics being part of the gradient update is also interesting.

I've also messed quite a bit with the distributed representations.

I'm not partial to any particular implementation. I'll use what works. That being said, I'm not armchair. I'll be backing this up with my own data as well.

I'm aware the tone was a little condescending and I won't take back what I said. I will actually back it up though ;).

My main point is just because something is controversial shouldn't stop you from trying it. That's what research is: trying new things.

> Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding.

Actually, I'm not sure we can do those things in quite 0.1 sec. All I know about this is from around minute 9 from http://www.radiolab.org/story/267176-never-quite-now/ . One guest on the show even estimates thinking the simplest thought to be on the order of 0.25-0.5 sec.

maybe concious thought, most of the stuff in the brain is on autopilot and happens behind our backs, conciousness gets the results and just builds narrative (often rationalizing the results to make us feel better)

Whelp, Page 54 blew my mind right out the window.

> E(hotter) - E(hot) + E(big) ≈ E(bigger)

> E(Rome) - E(Italy) + E(Germany) ≈ E(Berlin)

These things are linearly separable?!

Deep neural networks essentially transform input data into a vector space where the data is "easier" to model. So while the input vectors may not be linearly separable in the input space, the network learns how to transform the input vectors into a space where they are.

Yes. Hence 'miraculous' Word2vec.

These are great highlights, although your second point from [1] is a bit misleading. Humans have a 5% error rate because the average person can't differentiate between 150 different breeds of dogs and other similar nonsense.