After seeing an LLM do something like this, I've got to ask people who think that LLMs are just "stochastic parrots", "just predicting the next word", or are merely "a blurry jpeg of the web" to think about what's really going on here.
I'm not ruling out that this LLM output is "partially organic" rather than "fully regurgitated", but I'd be much more interested to see this LLM explain an obfuscated program that hasn't been floating around the Internet for 35 years.
They do much better with popular languages.
So, in other words, they perform precisely how you’d expect a stochastic parrot to perform?
The more popular the language the more likely the training corpus includes both very similar code samples and explanation of those code samples, and also the more likely those two converge on a “reasonable” explanation.
Ask it something it’s likely to have seen an answer for and it’s likely to spit out that answer… interesting? Sure, impressive? Maybe… but still pretty well captured by “a fuzzy jpeg of the web”.
Or exactly like you'd expect a human to perform.
Train a human mostly on English, and they'll speak English. Train them mostly on Chinese, and they'll speak Chinese.
Ahh, but ask a human a question in a language they don’t understand and they’ll look at you with bewilderment, not confidently make up a stream of hallucinatory nonsense that only vaguely looks statistically right.
> Or exactly like you’d expect a human to perform.
Not exactly, no… but with just enough of the uncanny valley to make me think the more interesting thought: are we really not much more than stochastic parrots? Or, in other words, are we naturally just slightly more interesting than today’s state of the artificially stupid?
I mostly agree with the stochastic parrot interpretation, but that doesn't undermine the usefulness or impressiveness. Even if it's just a highly compressed search index, that level of compression is amazing.
Start by find-and-replacing those #defines. You can iteratively deobfuscate things by hand. It's PITA and takes time, but it's doable.
If you hit a roadblock, run it in a VM.