Comment by rgoulter | Hacker News Reader

Comment by rgoulter | Hacker News Reader

Parent Full thread

With LLMs, I'd expect if something appears more frequently in the training data, then the LLM would be more effecient or smarter at dealing with it somehow.

Surely there's way more content related to programming in Python, JavaScript, Go, etc. than in Haskell.

So you'd expect some things like: an LLM is likely able to come up with an approach that's suited to Python/etc., and an LLM is likely able to work its way through Python/etc.

My experience has been: when using an LLM with a nice language (Nickel-lang, a modular configuration language with types and contracts) that it frequently guesses slightly wrong as to what works (e.g. guessing wrong about how stuff like { x = x + 1 } would work) that it spends more tokens than it otherwise might.

Ooh, you're saying you need to ask it twice to get the same real-world result. I thought you meant the tokenization was less efficient (as if `Endofunctor` were tokenized `Endo`+`Functor` while `AbstractFactory` was a single token or something like that)

This is an interesting hypothesis to test. Depending on the embedding model, such less frequently encountered character sequences in the coding domain could indeed produce less optimal tokens. A dedicated model trained from scratch on Haskell would be optimal but wouldn’t generalize well beyond that, but can serve as a deliberately overfitted baseline.

What I meant was really about the output tokens, on the API pricing level of interaction with an LLM.

Interesting, most functional languages are quite terse and heavily whitespace, so you would expect raw token counts like that to be lower simply because source documents are on average shorter. Even having to make up for disadvantage things like C++ style operators like &&= all having single tokens from heavy use across a variety of languages but fancy functional operators like <=> being much more rare and in worst cases of undertrained models needing multiple tokens to express.

But yes, terser source documents and low raw token counts could mean lower prediction rates and more prediction attempts needed to get intended results (again, especially if the model was undertrained in that particular language).

Reply on news.ycombinator.com