Tofu – Twitter bot

I'm not so sure it works well enough for us to be at all suspicious of that...

I just got my tofu tweet: "As a few words. rocking wearing? I injured my syntax highlighting =D? I gotta!"

It seems like a simple generator like http://www.wisdomofchopra.com/

These kinds of systems almost always use an n-gram model to generate text. It's a fascinating and surprisingly effective tool (the surprise is that implementing one is extremely simple).

Basically, it's based on collecting data from a corpus about probabilities of word transitions -- the probability of the "next word" given the n preceding words. So n-gram models often produce sentences that appear correct in small areas of a few words, but as a whole don't make much sense. The higher the "n", the more the generated sentences come to resemble the writing in the corpus. Once you get to quadrigrams, usually the model simply produces exact sentences verbatim from the text because the data becomes very sparse (how many times do you see "then I turned" in a corpus of tweets, for instance?), unless you have a very large corpus.

Like many NLP models, the n-gram model has seen many variations and tweaks that have differing levels of effectiveness, where often different tweaks produce more believable results for different corpora or variety thereof.

The math is quite simple. Let's take n=2, also called "bigrams", also called "Markov chains". In Pythonic pseudocode (because math notation doesn't work on HN) you could create a frequency distribution with:

  for i in range(1,words.length):

    model[words[i-1]][words[i]] += 1

assuming that "model" is a nested dictionary where keys have a default value of 0. then normalise:

  for prior,words in model:

    n = sum(words.values)

    words.values.map(lambda x: x / n)