Generating but not programming in them. They need to be reminded of the complete language definition they generated in every prompt, which is extremely costly, inefficient, and ultimately pointless, since any language you make up can't hold a candle to Python and its ecosystem, because it doesn't have an ecosystem, and the language itself doesn't exist in the training data.

How can you not get that? Do you believe LLMs remember what you show them between calls? That's not how they work. Each call starts from a clean slate, you have to re-describe the new language each and every call. There's no way to get around that. They are not magic. They do not learn from your prompts, which have absolutely no effect on the model itself.

If you think they do, you are falling for an illusion. ChatGPT is appending each of your incremental prompts to the full prompt, and it grows and grows longer and longer every time you add something. Sure, it summarizes when the full prompt gets to long, but that makes it distort and forget your language definition, and you have to add it again. If you give it the prompt to generate the language from scratch each time instead of the generated language itself, it generates a different language every time. You can't "cleverly hack" or "wish" your way out of that.

They may be good at generating new languages, but one thing that LLMs aren't good at apparently is warning you it's futile to generate a new language intended for llms to program instead of just using existing languages. They just play along and do ridiculous useless things out of syncophancy.