Let's not underestimate LLM's ability to do in-context learning. Perhaps it can just read the new lang's docs and apply what it already knows from other languages
So why would they be able to "read" the docs and use that knowledge except up to pattern matching level. That's why I also assume, that tons of examples with results would do better than lang docs, but I haven't tested it yet.
So while they cant learn arithmetic they should be able to learn programming languages given that they are way closer to what it was designed and trained for.