This comment is wrong in two ways:

1. Current LLMs do much better than produce "small programs that already exist in multiple forms in the training data". Of course the knowledge they use does need to exist somewhere in training data, but they operate at a higher level of abstraction than simply spitting out programs they've already seen whole cloth. Way higher.

2. Inventing a new compression algorithm is beyond the expectations of all but the the most wild-eyed LLM proponents, today.