Hacker News Reader

Comment by augment_me | Hacker News Reader

What I feel a bit annoyed by, and what I feel obviously LLM-run ablations like this fail to capture, is any kind of reflection around previous research or any kind of proof that this is the best you can do. You don't know, you pulled the lever and you got something, is the best? Can you do better? What is the constraint?

As an individual researcher, you do not have 22M$ to run a massive brute-force search for your problem. You are constrained to your little subscription and you will barely dip your toe in the sea of possible solutions to a problem. So letting Claude run an autoresearch loop on your problem and then having it summarize it for you brings 0 value because you dont know what the downsides and trade-offs of LRU caches were, and how you would possible solve it.

View on HN

I don't think the problem is a lack of previous research, the problem is it looks like the target metrics were selected by an LLM, and it's unclear what the LLM was told to optimize or if it was just told to try and make a better KV-cache.

What I've noticed with Claude is that regardless of how I prompt, it will find a few metrics to optimize. Often the metrics it chooses to optimize have zero relation to the actual metrics I want to optimize, which are ones that cannot be measured without more work than Claude can do in a single 1M token context window. It's really hard to stop Claude from optimizing whatever metrics it can find when the actual metrics I want to optimize are not computable.

Sometimes it's good enough to acknowledge a correct answer, and slowly build intuition towards it. Academic problem solving is rarely a straight path, anyway.

> What I feel a bit annoyed by, and what I feel obviously LLM-run ablations like this fail to capture, is any kind of reflection around previous research or any kind of proof that this is the best you can do.

At least the former is easy: Claude is very happy to do a prior art search and pretty good at it, too.

you can just ask it to explain those things!

Sure, and it will explain it to you from the same embedding space that yielded the solution that LRU cache is unbeatable.

It'll just be telling you what the data it ingested claims. Not what is necessarily true. It's as subject to garbage in, garbage out as anything, and you don't know what it's actually trained on.

Reply on news.ycombinator.com