Hacker News Reader

Comment by amelius | Hacker News Reader

Shouldn't the LLMs therefore train on code where the variable names have been randomized?

Perhaps it will make them more intelligent ...

No. Variable names contain valuable information. That's why humans use them too.

AIs are finite. If they're burning brainpower on determining what "x" means, that's brainpower they're not burning on your actual task. It is no different than for humans. Complete with all the considerations about them being wrong, etc.

Training them on randomized var names doesn't mean you should do it deliberately during inference ...

Also, I think this is anthropomorphizing the llms a bit too much. They are not humans, and I'd like to see an experiment on how well they perform when trained with randomized var names.

Neither "variable names contain valuable information" nor "AIs are finite" are anthropomorphization. That variable names contain information is not only relative to some sort of human cognition; they objectively, mathematically, contain information. Remove that information and the AI is going to have to reconstruct it (or at least most of it), and as finite things, that's going to cost them because nothing like that is free for finite things. None of my assertions here depend on the humanity, the AI-ness, the Vulcan-ness, or any other conceivable finite intellectual architecture of any coding agent. It only depends on them being finite.

Let's stop with the comparison to humans, I'm more interested in why it would hurt to train LLMs with harder puzzles. Isn't that what we're doing all the time when training llms? I'm just suggesting an easy way to construct new puzzles: just randomize the varnames.

An even easier way to construct new puzzles is to fully randomize the problem statements and intended solutions.

When you take out the information from the variable names, you're making the training data farther from real-world data. Practicing walking on your hands, while harder than walking on your feet, won't make you better at hiking. In fact, if you spend your limited training resources on it, the opportunity cost might make you worse.

But we know that variable names do not matter whatsoever to a compiler. Now I do agree with you intuitively that LLMs perform better on meaningful variable names, without looking at hard data - but I don't think it has anything to do with "brainpower" - I just think your input data is more likely to resemble training data with meaningful variable names.

I think you're just arguing semantics. It seems intuitively obvious that if I have some simple physics code

    newPosition = currentPos + velocity * deltaTime

and change it to

    addressInChina = weightByGold + numberOfDogsOwned * birdPopulationInFrance

that both a human and likely an LLM will struggle to understand the code and do the the right thing. The thing we're discussion is does the LLM struggle. No one cares if that's not literally "brain" power. All they care about is does the LLM do a better, worse, or the same

> I just think your input data is more likely to resemble training data with meaningful variable names.

Based on giving job interviews, cryptic names are common.

Exactly because the task is harder if the variable name does not contain any information is what makes training like that a good idea. It forces the LLM to pay attention to the actual code to get it right, which in training is a Good Thing (TM).

Right. Inherent information complexity goes down as all metadata is stripped from the variable name and the value has to be re-contextualized in a fresh logical chain every time.

LLMs do not have brains and there is no evidence as far as I know that they "think" like human beings do.

LLMs do not reason at all (i.e. deductive reasoning using a formal system). Chain of thought etc simulate reasoning by smoothing out the path to target tokens by adding shorter stops on the way.

Do you reason? "LLMs do not reason at all" casts that into doubt immediately.

That being true does not mean that there are no limits to whatever it might be doing, which might be wasted with ambiguous naming schemes.

I am far from an AI booster or power user but in my experience, I get much better results with descriptive identifier names.

LLMs are only capable of performing a finite amount of computation within a single forward pass. We know that much.

They are also known to operate on high level abstracts and concepts - unlike systems operating strictly on formal logic, and very much like humans.

LLMs do see randomized identifiers, whenever they encounter minimized code. And you can get a bit of an idea how much they learn, by giving an LLM some minimized JS and asking it to restore it with meaningful var names.

When I tried it once the model did a surprisingly good job, though it was quite a while ago and with a small model by today's standards.

No, they're more likely to predict the correct next token the closer the code is to identical to the training set, so if you're doing something generic short names will get the right predictions and if you're doing something in a problem domain, using an input that starts the sequence generation in a part of the model that was trained on the problem domain is going to be better

They're trained on plenty of code with bad variable names.

But every time you make an AI think you are introducing an opportunity for it to make a mistake.

if you train them on randomized names, they'll also suggest them.

better to not, I think.

Reply on news.ycombinator.com