Hacker News Reader

Comment by tshadwell | Hacker News Reader

I'm not sure how "unreasonable" the effectiveness of RNNs are if the corpus output at 2000 iterations isn't significantly better than a simple prefix based markov chain implementation [1] (and for the regular languages, with some extra bracket-checking), but I found the evolution visualizations really interesting.

[1] http://thinkzone.wlonk.com/Gibber/GibGen.htm

View on HN

It's quite unreasonable. He could have optimized it more for fooling humans in Gibberish generation, but that would not show the general effectiveness of the approach. The power shows (quantifiably) in compression: 1.57 bits per character wikipedia is quite hard to beat. Of course, Markov Chains are essentially universal models, so the training algorithm is the crucial distinction.

I believe Markov Chains as a model quickly become inefficient (specially memory-wise) as you increase the complexity (long range correlations) of your prediction. It's an unnecessarily restrictive model for high complexity behavior that state of the art RNNs skip entirely.

The state of the art in compressing wikipedia is 1.278bits (on a certain subset) [1]. So that does seem pretty good.

[1] http://prize.hutter1.net/

Except this NN isn't really a compression of Wikipedia since it can only generate Wikipedia-like nonsense.

It's a compression of Wikipedia in the sense that the NN generates probability estimates of the next character given the previous; the gibberish is simply greedily asking the NN repeatedly what the most-likely next character is. However, plug it into an arithmetic coder and start feeding in an actual Wikipedia corpus, and hey presto! a pretty high performance Wikipedia compressor, which works well on Wikipedia text but not so well on other texts (like this one, with its lack of brackets).

There's very little difference between a contextual predictive model like this and the guts of a compressor.

If your prediction is good enough that you can always come up with two possible predictions for each character, each of which has a 50% chance of being correct, then obviously you can compress your input down to one bit per character by storing just enough information to tell you which choice to pick. More generally, you can use arithmetic coding to do the same thing with an arbitrary set of letter probabilities, which is exactly what you get as the output of a neural network.

When the blog post says the model achieved a performance of "1.57 bits per character", that's just another way of saying "if we used the neural network as a compressor, this is how well it would perform."

I'd be interested in seeing this NN perform a lossless compression of Wikipedia at 1.57 bits per character.

That Markov Chain model operates on 4-grams by default. The RNN featured in the article generates output character-by-character, which is significantly more impressive. Here's a sample from the Markov Chain model operating on 4-grams:

  Ther deat is more; for in thers that undiscorns the unwortune, 
  the pangs against a life, the law's we know no trave, the hear, 
  thers thus pause.

The only reason why it seems like the model can occasionally spell, and create anglo-sounding neologisms, is because it operates on 4-grams.

Here's some character-by-character output from the same Markov Chain model.

  T,omotsuo ait   pw,, l f,s teo efoat t hoy tha fm nwo   
     bs rs a h enwcbr lwntikh  wqmaohaaer ah es aer 
  mkazeoltl.etnhhifcmfeifnmeeoddssmusoat irca   
  do'ltyuntos sih i etsoatbrbdl

"do'lty untos sih i"

maybe the computer was drunk?

it's a completely legit invocation for awakening cthulhu.

It balances parentheses and keeps track of other long range dependencies, something markov chain implementations cannot do.

Welcome to the unbearable forced-ness of titles. Everyone's making a nod to Milan Kundera these days.

http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_...

First, it's not a nod to Kundera, but to a classic math related work that predates Kundera's book.

Second, even if it was, really? As if we see plays on Kundera titles regularly on the web?

I doubt it's a reference to Kundera.

I was thinking that both Eugene Wigner's 1960 article 'The Unreasonable Effectiveness of Mathematics in the Natural Sciences'[0] and Karpathy's 'The Unreasonable Effectiveness of Recurrent Neural Networks' probably touch deep aspects of the nature of existence. The first on why the universe exists and is mathematical - because at the fundamental level it is mathematical[1], and in Karpathy's case the RNNs are probably effective because they are close to the mechanisms of human consciousness.

[0] Wigner's article: http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html

[1] 'physical world is completely mathematical' theory: http://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_...

Reply on news.ycombinator.com