Hacker News Reader

> GPT-2 is not just a dictionary, it's enormous non-linear function.

Dictionaries are non-linear functions.

> This compressor gets compression levels which are unachievable with a dictionary.

I don't see why this is the case, care to elaborate?

A dictionary doesn’t give a prediction about what the next word in string will be. GPT-2 does and those predictions can be used so save space.

A dictionary of n-grams does, which I think is what he is describing.

GPT is way better at next-word predictions than n-grams. However, I agree with OP that the resulting compression scheme is probably not very practical all things considered.

A dictionary of n-grams does not change its probabilities based on "text seen so far", so you'd need an unbounded n. In practice, n-gram dictionaries don't encode for more than very low values of n, so while yes, abstractly we can say that GPT-2 is equivalent to an n-gram dictionary with some high n, those dictionary cannot be constructed, let alone used, leaving the equivalency is purely theoretical.

GPT-2 is not a dictionary.

>GPT-2 is not a dictionary.

I was only responding to the statement that dictionary methods don't give a prediction of the next word in a string will be. I wasn't in any way saying GPT-2 is a dictionary.

Reply on news.ycombinator.com