Whereas GPT2 hallucinated random stuff about the US government. A text prediction model should predict what the user wanted to type, so if you evaluate the models based on that, GPT2 actually performed horribly, since the user showed zero intent in talking about the US.
The default goal everyone is assuming is spitting out the longest correct sequence possible.
But in reality the mental cost of a wildly wrong prediction is much higher than the mental cost of a slightly wrong one, so what you'd train the model for is sequences of a few words at most being with higher confidence.
Most people can/will tune out slightly wrong words especially as they get a feel for what autocorrect is good and bad at.
If you unleash the full range of tokens GPT 2 can normally output, you'll constantly be blasting out words they didn't expect.
—
The fact your long sequence prediction got better doesn't matter because the UI is autocomplete not "auto-write": they're still expecting to drive, and a smart but noisy copilot is worse than a dumb but lazy one in that case.
I wouldn't be surprised if they trained the model to an effective context window of just a few hundred tokens with that in mind
Too many people are focused on technology for technology’s sake.
"Today is a good day for you to be able to do it more than i just a few weeks to get a new."
Or maybe I wanted to express myself in the timeless words of the poets:
"A day, a day of glory! A day that ends our woe! A day that tells of triumph. Against our vanquished foe!"
"Rose is a rose is a rose is a rose. Loveliness extreme. Extra gaiters. Loveliness extreme."
"A-well now, everybody's heard about the bird, everybody's heard about the bird, About the bird, the bird, bird bird bird, Haven't you heard about the bird? Don't you know that the bird's the word?"