Hacker News Reader

Comment by abeppu | Hacker News Reader

but the experiments it did that "improved" validation BPB in the GH screenshot were all basically hyperparameter changes right? So is this better or worse, either per experiment or per unit time, than hyperparameter tuning techniques that don't involve an LLM? It's not clear from this if the LLM is more or less making random changes which sometimes work , and or the LLM thinking actually finds "good" changes because of what the LLM has internalized. E.g. how does this compare to a hyperparameter tuning pass with e.g. BayesOpt that does the same number of 5-min training experiments?

View on HN

this is very far from hyperparameter tuning in at least three important ways:

- it can modify code arbitrarily, the notion of a "hyperparameter" dissolves

- there is no need to run "sweeps" - this is the standard parallel process that wastes compute. because LLM agents are sequential, they can do more efficient versions such as binary search to narrow in on the right setting very quickly (usually many parameters will have a U shaped optimal setting).

- it's fully automatic, it doesn't require human in the loop to mess with the code.

You're right that many of the changes it seems to make out of the box (as I intentionally did not try to prompt engineer it too hard yet because I was curious what you get by default) seem to be tuning existing hyperparameters. not all of the changes are like that - e.g. it tried to replace the non-linearity, etc. I will say that overall (and again, out of the box) the LLM feels unwilling to creatively pursue a research direction or something like that. The models feel very "cagy" and "scared" when they are given problems that are a little too open ended. But that's just where the fun parts, e.g. I had some early successes with the idea of a "chief scientist" that was basically a never-ending plan mode that looked at what worked, didn't work, tried to find related code/papers, and created a long list of experiments to try, which it could then send to junior engineers running in tmux sessions. I think quite a few approaches are possible, so I think it's a nice canvas. The reason we're not getting "novel research" feels like half capability issue and half skill issue.

On the skill side, personalities could be fun:

"You are Yann Lecun's last PhD candidate, and he hates you and you hate JEPA. You are determined to prove that a non-world model can reach AGI. In order to get your PhD you have to be creative and come up with new ideas. Remember without it, you're stuck."

Seems like the best way to reach AGI is to give LLMs anxiety.

How about the very last "Kept Improvement" in the plot? It's titled "random seed 42 -> 137". I do think this project is quite conceptually interesting, but the model literally choosing a different random seed to achieve lower loss feels pretty far removed from the flowery sci-fi writing at the top of the readme.

The 42 -> 137 also jumped out at me. On the face of it, the associated improvement sure does sound like overfitting to the eval set.

It shows that both Karpathy and the LLM have good taste in random seeds: the answer to life, the universe and everything, and ~1/(the fine structure constant)

So the interesting part about this one is that when I had the model write up the results for that session:

https://github.com/karpathy/autoresearch/discussions/32

Look at its comment about this "improvement":

""" Surprising non-results:

- Changing random seed from 42→137 improved by 0.0004. Seed 7 was worse. Make of that what you will. """

So the model knows! It knows that this is a weird thing to do after the fact. I think it's silly that the model even tried and that it ran this, but some part of it also knows that it was wrong. This means that this is fixable by prompt.md

The disposition problem you describe maps to something I keep running into. I've been running fully autonomous software development agents in my own harness and there's real tension between "check everything" and "agent churns forever".

It'a a liveness constraint: more checks means less of the agent output can pass. Even if the probabilistic mass of the output centers around "correct", you can still over-check and the pipeline shuts down.

The thing I noticed: the errors have a pattern and you can categorize them. If you break up the artifact delivery into stages, you can add gates in between to catch specific classes of errors. You keep throughput while improving quality. In the end, instead of LLMs with "personas", I structured my pipeline around "artifact you create".

I wrote up the data and reasoning framework here: https://michael.roth.rocks/research/trust-topology/

Reply on news.ycombinator.com