Hacker News Reader

Comment by ip26 | Hacker News Reader

The language doesn’t just map to English, it allows high level concepts to be expressed tersely. I would bet it’s much easier for an LLM to generate python doing complex things than to generate assembly doing the same. One very simple reason is the context window.

In other words, I figure these models can benefit from layers of abstraction just like we do.

View on HN

It allows these concepts to be expressed legibly for a human. Why would an ai model (not llm necessarily) need to write say "printf"? It does not need to understand that this is a print statement with certain expectation for what a print statement ought to behave as in the scope of the shell. It already has all the information by virtue of running the environment. printf might as well be expressed as some n-bit integer for the machine and dispense with all the window dressing we apply when writing functions by humans for humans.

Because there's a lot of work behind printf that the llm doesn't need or care to reproduce

You're not just using the language, but all of the runtime and libraries behind it

Thinking it's more efficient for the llm to reinvent it all is just silly

Indeed. And aside from that, LLMs cannot generalise OOD. There's relatively little training data of complex higher order constructs in straight assembly, compared to say Python code. Plus, the assembly will be target architecture specific.

Right and all of that in the library is built to be legible for the human programmer with constraints involved to fit in within the syntax of the underlying language. Imagine how efficient a function would be that didn't need all of that window dressing? You could "grow" functions out of simulation and bootstrapping, have them be a black box that we harvest output from not much different than say using an organism in a bioreactor to yield some metabolite of interest where we might not know all the relevant pieces of the biochemical pathway but we score putative production mutants based on yield alone.

“printf” is an n-bit integer already. All strings are also numbers.

I completely understand what you are saying but ip does make an interesting point.

Why would chain of thought work at all if the model wasn't gaining something by additional abstraction away from binary?

Maybe things even go in the other direction and the models evolve a language more abstract than English that we also can't understand.

The models will still need to interface though with humans using human language until we become some kind of language model pet dog.

Reply on news.ycombinator.com