Autoresearch on an old research idea
ykumar.me
ykumar.me
I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are very poorly maintained / niche libraries that have quite a lot of content written about them but what I can only imagine is very limited use in real production environments.
On the other hand, we have domain expert “consultants” in our leadership’s ears making equally absurd recommendations that we constantly have to disprove. Maybe an agent can occupy those consultants and let us do our work in peace.
A lot depends on whether it is expensive to you. I use Claude Code for the smallest of whims and rarely run out of tokens on my Max plan.
Steelmanning your question though, I guess you could come up with some sort of tiered experimentation scheme where you slowly expose it to more data and more compute based on prior success or failures.
For all the folks spending a lot of time and energy in setting up MCP servers, AGENTS.md, etc. I think this represents more that the LLM cannot do what it is being sold as by AI boosters and needs extreme amounts of guidance to reach a desired goal, if it even can. This is not an argument that the tech has no value. It clearly can be useful in certain situations, but this is not what OpenAI/Anthropic/Perplexity are selling and I don’t think the actual use cases have a sustainable business model.
People who spend the energy to tailor the LLMs to their specific workflows and get it to be successful, amazing. Does this scale? What’s going to happen if you don’t have massive amounts of money subsidizing the training and infrastructure? What’s the actual value proposition without all this money propping it up?
I found LLMs make a fabulous frontend for git :-D
This was the case for me a year ago. Now Claude or Codex are routinely delivering finished & tested complete features in my projects. I move much, much faster than before and I don’t have an elaborate setup - just a single CLAUDE.md file with some basic information about the project and that’s it.
What’s the point of adding features that are inscrutable? I have gotten Claude to make a feature and it mostly works and if it doesn’t work quite right I spend a massive amount of time trying to understand what is going on. For things that don’t matter too much, like prototyping, I think it’s great to just be able to get a working demo out faster, but it’s kind of terrifying when people start doing this for production stuff. Especially if their domain knowledge is limited. I can personally attest to seeing multiple insane things that are clearly vibe coded by people who don’t understand things. In one case, I saw API keys exposed because they were treating database users as regular user accounts for website login auth.
> I move much, much faster than before
This is a bad metric as has been attested multiple times in unrelated situations. Moving faster is not necessarily productivity nor is it value.
You are assuming that the additional speed comes at a cost of codebase comprehension. For me it's not the case - I never push generated code I don't fully understand. It does take time, sure, but it still takes me much less time to write a spec, execute with AI and then review than write the thing myself.
The irony is that "extreme amounts of guidance" is exactly what makes a human domain expert valuable, too. A senior consultant isn't smarter than a junior one; they have a better methodology for directing attention to what matters. The actual problem with the "just throw an agent at it" approach isn't cost. It's that without structure, you can't tell the 10% of useful output from the 90% of noise
This is, of course, only applicable if doing a single test is relatively fast. In my work a single test can take half a day, so I'd rather not let an agent spend a whole night doing a bogus test.
Definitely not in the budget for non-VC-backed companies who aren’t in the AI bubble.
So this may be only temporarily unavailable for many.
I've been thinking of this as a gap between VC/Kickstarter and just doing it yourself. Most early ML experiments are too small for formal funding but too expensive to casually self-fund. So I built ML Patron where anyone can chip in a few bucks to sponsor an experiment they're curious about. I honestly don't have a good answer yet for how this turns into returns for sponsors in a traditional business sense. For now it's just open research patronage, like "I'd pay to know the answer to this". Platform runs it on cloud GPUs with public MLflow tracking.
Still very early: https://news.ycombinator.com/item?id=47563959.
Why is that?
I don't doubt you, but when Shigeo Shingo created SMED (Single Minute Exchange of Die), die changes were an hours long process.
- https://arxiv.org/abs/2602.02660 (MARS)
- https://arxiv.org/abs/2601.14525 (Execution-grounded automated AI research)
- https://arxiv.org/abs/2601.10402 (ML-Master 2.0)
Good lens.
The crux of the auto research repo is basically one file - program.md which is a system prompt that can be summarized as “do this in a loop: improve train.py, run the training, run evals, record result. Favor simplicity”. The other files are an arbitrary ML model that is being trained.
This has been the standard approach for more complex LLM deployments for a while now in our shop.
Using different models across iterations is also something I've found useful in my own experiments. It's like getting a fresh pair of eyes.
Alternatively, a modular model with multiple “experts” that I could mix and match for my specific stack
I don’t need the model to know all of the Internet plus 20 different human languages. I just want it to be really good with the stack of the project
[1] https://github.com/ykumards/eCLIP/commits/main/autoresearch
i.e. perhaps minimal changes to autoresearch can take control for cost-effective research to occur.
If you look at the commits, you can see that all it does is just set different values for different parameters of continuous values: the type of thing that I trust statistics a lot more than reasoning. Optuna can make very informed decisions when making lots of different changes at once, slowly converging towards optimal parameters, where the LLM seems to be throwing stuff at a wall and see what sticks.
What would work best if the LLM would try to approach things on a higher level, ie use Optuna, but reason about better approaches for algorithms and/or data or whatever. But what it ends up doing is tuning parameters manually, only one / a few at a time, extremely inefficient and unlikely to be optimal.
> Yes but at that point you may as well use a proper hyperparameter tuning framework like optuna if all the LLM agent is supposed to do is do hyperparameter tuning.
while the "novelty" of autoresearch is that it may symbolically reason about the computation, analyze the codebase, etc. i.e. a wider search space (harder) but symbolic reasoning.
For a fixed search space it will almost certainly be better though.
The bottleneck in AI/ML/DL is always data (volume & quality) or compute.
Does/can Autoresearch help improve large-scale datasets? Is it more compute efficien than humans?
There always are. You need to think about what those would be, though. Autoresearch outsources the thinking to LLMs.
Non-parametric optimization is not a new idea. I guess the hype is partly because people hope it will be less brute force now.
I recall reading about a stochastic one years ago: <https://github.com/StanfordPL/stoke>
There are lots of old ideas from evolutionary search worth revisiting given that LLMs can make smarter proposals.
Yes, for example "swarm optimization".
The difference with "autoresearch" (restricting just to the HPO angle) is that the LLM may (at least we hope) beat conventional algorithmic optimization by making better guesses for each trial.
For example, perhaps the problem has an optimization manifold that has been studied in the past and the LLM either has that study in its training set or finds it from a search and learns the relative importance of all the HP axes. Given that, it "knows" not to vary the unimportant axes much and focus on varying the important ones. Someone else did the hard work to understand the problem in the past and the LLM exploits that (again, we may hope).
Years ago there were big hopes about bayesian hyperparameter optimization, predicting performance with Gaussian processes etc, hyperopt library, but it was often starting wasteful experiments because it really didn't have any idea what the parameters did. People mostly just do grid search and random search with a configuration that you set up by intuition and experience. Meanwhile LLMs can see what each hyperparameter does, it can see what techniques and settings have worked in the literature, it can do something approximating common sense regarding what has a big enough effect. It's surprisingly difficult to precisely define when a training curve has really flattened for example.
So in theory there are many non-LLM approaches but they are not great. Maybe this is also not so great yet. But maybe it will be.
Not true at all. The whole point of ML is to find better mappings from X to Y, even for the same X.
Many benchmarks can’t be solved by just throwing more compute at the problem. They need to learn better functions which traditionally requires humans.
And sometimes an algorithm lets you tap into more data. For example transformers had better parallelism than LSTMs -> better compute efficiency.
So the bottleneck was compute. Which is compatible with 'data or compute'. But to accept your point, at the time the algorothmic advances were useful/did unlock/remove the bottleneck.
A wider point is that eventually (once compute and data are scaled enough) the algorithms are all learning the same representations: https://arxiv.org/pdf/2405.07987
And of course the canon: https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat... http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Scaling compute & data > algorithmic cleverness
I bring up transformers because scaling compute and data was unlocked by a better algorithm. It matters a lot because scaling compute isn’t always an option.
Most of the gains came from fixing a bug + hyperparameters with optuna which is supposed to be already quite automatic (you set the list of all the var with values you want to try and voilà). I guess a simple claude code session would fix that in a few minutes instead of a full day.
To me, I guess the main value of Autoresearch would be to test different kind of architectures. It's sometimes hard to know what to choose and it would probably give a nice overview.
Anyone used it for exploratory modeling?
That's such a weird switch. There's lots of free medical imaging online. Example: https://www.cancerimagingarchive.net/
To get CLIP to work properly we typically need large batch sizes. So the experiments in the original paper were quite heavy, and ran parallel across 8 GPUs.
I wrote up some more notes on that here: https://simonwillison.net/2026/Mar/13/liquid/
It’s certainly cool, but the optimizations are so basic that I’d expect a performance engineer to find these within a day or two with some flame graphs and profiling.
So cheaper than a performance engineer for a day or two... but the Shopify CEO's own time is likely a whole lot more expensive than a regular engineer!
What about more distant software projects? Give it the CPython source code and say you want it to be faster.
I used it on a data science project to find the best rules for achieving a defined outcome. At first, for fun, then I actually used some of its insights (and it caught a sampling issue I overlooked, oops)
The author doesn't really describe which part was a slog, I thought autoresearch was supposed to be pretty much set and forget.
How does one run Claude Code without network access?
in both cases you'd limit it so CC can only talk to the required Anthropic APIs.
So not zero access, but as close to it as you can get.
The docker container didn’t have network access. Claude didn’t have permission to execute anything other than the run.sh bash script, which would orchestrate the docker run
thanks for sharing your results and the road to them!
I started looking at Kaggle again and autoresearch seems to converge to many of the solution vibes there.
Wild ensembles, squeezing a bit of loss out. More engineering than research IMO
If you're resource unconstrained then BO should ofc do very well though.
Good Bayesian exploration is much, much better than grid search, and does indeed learn to avoid low value regions of the parameter space. If we're talking about five minute experiments (as in the blog post), Bayesian optimization should chew through the task no problem.