Hacker News Reader

Comment by swatcoder | Hacker News Reader

Yes.

The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.

The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.

View on HN

At work i only had early copilot which was hysterically bad at everything. As i wanted it to do the same task repeatedly and could spot wrong results instantly i kept evolving a prompt that attempted to correct all ways it found to do it wrong. It kept inventing new ways to get it wrong until it eventually got it right 90% of the time. My theory is that an avanced model that has no issues with a task could do the prompt enginering much better than i ever could. You could for example run x different queries that all do the same thing and compare the results y times. If there are >1 correct results and the wrong versions are all unique you should be able to drill down to a valid result with even a truly shit model running on a potato. Basically what humans do.

Early Copilot was tab-complete in editors and was honestly the best version of LLM-assisted development I've used, because it was intentionally small in scope, trivial to verify output from at a glance and easy to opt in and opt out of. I have yet to see anything more useful in terms of code generation; very small-scale code generation (think function-level) comes close, but is a lot more tedious.

For analysis, bug hunting, overview and some porting work to popular languages and so on I think the current SotA is fantastic, but they're still very disappointing for code generation past function or small module level.

> "I have yet to see anything more useful in terms of code generation"

Recently my employer hooked Claude into a bunch of SaaS services like a ticketing system and an asset inventory system, and I feel a big productivity boost is that I don't have to dig for their open browser tabs, remember their product name/URL/where I put the bookmark, find that my sessions have logged me out "for my protection" since I last looked, find my MFA code, approve sign-in on my phone, then use their mismashed/sluggish/poor-UX interfaces and slow searches to loko for things, and can instead ask Claude "search these systems for anything about <issue>" and it just does. It correlates between systems, summarizes things, and gives me references which system and where to look to check what it found. Internal search on a mess of acquired and disjoint systems.

This feeling reminds me of two classic Joel Spolsky blog posts below, and your comment here about the surprisingly useful tab-complete gives me a similar feeling. I wonder how many smaller candidates there are where LLMs are more helpful and less hyped, against the big vibe-coding solve-everything hype which are currently less helpful than their hype?

References [1]: "A lot of us thought in the 1990s that the big battle would be between procedural and object oriented programming, and we thought that object oriented programming would provide a big boost in programmer productivity. I thought that, too. Some people still think that. It turns out we were wrong. Object oriented programming is handy dandy, but it’s not really the productivity booster that was promised. The real significant productivity advance we’ve had in programming has been from languages which manage memory for you automatically. [...] Whenever you hear someone bragging about how productive their language is, they’re probably getting most of that productivity from the automated memory management, even if they misattribute it. Sidebar: Why does automatic memory management make you so much more productive? 1) Because you can write f(g(x)) without worrying about how to free the return value from g, which means you can use functions which return interesting complex data types and functions which transform interesting complex data types, in turn allowing you to work at a higher level of abstraction"

and [2]: "I have seen many language and programming fads come and go. But there’s only ONE, that’s right, ONE language feature I’ve ever seen that actually improves your productivity significantly. No, it’s not object oriented programming; no, it’s not intentional programming or assertions or programming by example or CASE or UML or XML or Java. The only thing that improves your programming productivity is using managed code – that is, using a language in which memory management is automatic."

[1] https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost...

[2] https://www.joelonsoftware.com/2001/10/17/working-on-citydes...

we could have had that before, but in past era of tech companies, APIs were not designed with an individual needs in mind and management was all too happy to order you to update status (manually) across JIRA, Slack and whatever other tracking system they were using.

I think this sounds fantastic, and I welcome all developments in these types of areas where you lean heavily on the hyper-attentive, very fast assistant aspect of LLMs intended to make information more easily searchable, visible and the results more accurate.

I agree that the mismatch between the expectations of OOP as a paradigm and this situation right now feel very similar. I suppose the many billions pumped into selling vibecoding and large-scale code generation via LLMs of other kinds is sort of like the massive amount of money pumped into marketing Java as a viable alternative long before it actually was.

Just like with human langages their only advantage is the community around it, the culture. (Im pretending size is a part of that Obj) No one does it but that is how you should pick it.

I've done this. Ask Claude to permutate a prompt and run Claude Code or a subagent to observe effects until success. Claude will iterate on the prompt faster than I could and likely catch more edge cases, too. It's one of those tasks where the end goal is very clear and the agent just needs to iterate on permutations. Arguably this is a perfect match, too, because prompt engineering is really just text generation, so why not have a text generation machine do it.

Trying until you find something that doesn’t fail is the model used throughout evolution & engineering, think of bridges collapsed, airplanes falling from the sky or exploded steam locomotives.

In engineering you learn from these mistakes and try never making them again. Do we want to go through this evolution every time we solve a software issue? Just because we can with an unlimited number of cheap tokens? I think not, I’d rather use the knowledge build up that also knows about the edge cases forgotten to test. Or better, use multiple models that evaluate each other, as Entropic describes it in their recent report https://news.ycombinator.com/item?id=49316271

> We expect that agents coordinating in the wild will act in higher variance ways than we see here, because they’ll have different backgrounds and therefore different contexts. They also, presumably, won’t all be Claudes.

Yes! As long as you have some criteria to judge the final answer, you can do a kind of "prompt-side RLVR", where you have the model generate prompt changes, try a bunch of different prompts and see which ones improve the results.

You don't necessarily need a bigger model to do this.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Recent comment touching on this in relation to LLM's in more depth: https://news.ycombinator.com/item?id=49322695#49323341

The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true.

Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.

This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess.

I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.

The Bitter Lesson says that the only things that scale are search and learning.

Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson.

In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them aren't designed to do chess things, they're designed to do autoregressive token decoding, which isn't a search process. Reasoning traces can resemble a search process, but they're far less efficient - the LLM would have to work out each legal move, test each one, calculate a score, and simulate minimax over all of that. Assuming the LLM is smart enough to even do all that.

A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.

> A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.

Now let's look at the bitter lesson again. It says that general methods that leverage computation are ultimately the most effective, and by a large margin.

That's different from just saying to leverage computation (which is how I would interpret "unlocks more search/learning"). If the lesson is "more computation wins, when sufficiently channeled" you're basically looking at a truism. Of course more computation beats less when it's used right. The bitter lesson is about abandoning specialization in order to get more computation, and while there's a couple ways where that helps with chess, there's a lot more ways where it's counterproductive. It looks like it's more true for Go than it is for chess, and that it's not universally true. It probably correlates with the state space.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

> a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games

Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:

> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]

and

> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]

The actual bitter lesson is this:

> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]

Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.

Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000

This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does.

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]

This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks? Not the point or the disappoint. It's always neats and scruffies fighting about using some kind of recognizable method (logic) instead of magic (ML).

> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.

More to OP's point I think: nowadays when someone wants to beat you over the head with the bitter lesson, they aren't as careful to include learning and search. They want to say learning leads to intuition (magic) whereby we can avoid work (logic/search), and maybe argue or assume from there that neats and scruffies is settled. TBF, something like reasoning in latent space does resemble intuition!

But the real lesson is confirmed every time we bother to check, and not very bitter for anyone. Search/learning/logic are ALL always necessary on any sufficiently difficult problems, and hybrids that interleave always outperform everything else. Stockfish being the example in this thread that different camps of absolutists would like to claim, but also all the MCTS examples, evolving examples, and new hybrids all the time. My favorite lately: https://arxiv.org/pdf/2511.08983

There's two different goals to AI research - one was to get results - a chess engine thst wins, etc. But the other goal (which seems to have been abandoned in the deep learning era) was to use AI to help understand how human minds work. A chess engine modeled after human grandmasters is much more interesting in that regard than either a min-max algorithm like beat Kasparov or modern deep learning engines.

> This was/is basically a strawman though. Like maybe "human input winning" was desirable for chess masters but for computer science wonks?

Oh no!

The whole field was full of people whose entire career was built around the idea of developing smart priors.

To quote Wikipedia:

> For computer vision in particular, much progress came from manual feature engineering, such as SIFT features, SURF features, HoG features, bags of visual words, etc. It was a minority position in computer vision that features can be learned directly from data

This undersells the change though! David Lowe's reputation as the best image researcher in the world was based on his SIFT patent[1]

This approach worked until 30 September 2012.

That was a bitter day for many, many computer science researchers.

[1] https://en.wikipedia.org/wiki/Scale-invariant_feature_transf...

Is being wrong/ignorant about whether/how something can be automated the same as having a preference for doing it manually? Maybe so if it's your patent, your thesis I guess..

But as it relates to more/less magic, maybe the more modern lens on this is e.g. https://arxiv.org/html/2505.11581v1 . Is manual feature-engineering more like what you'd evolve, or more like what you'd get from SGD ? Feasibility and performance is always a question, there are others like what is robust, stable, adaptable, predictable, explainable. Maybe the manual-features people were interested in something besides the manual part? Maybe the story isn't so simple, and maybe it's not finished yet.

I think you have it backwards.

The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.

Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.

Also, training it on chess books is literally training it on human knowledge, and not the actual game, which is exactly what the bitter lesson says not to do.

> I think you have it backwards.

> maybe if we use a blend of raw data and hand-crafted heuristics

I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.

I was replying to this:

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games.

If you can draw any lessons from chess commentary, I think it’s very reasonable to call it “hand-crafted heuristics.”

Hand-crafted even if you're feeding in the raw commentary? That seems like a weird way to consider it. Wouldn't that make LLMs in general "hand-crafted"?

And raw games plus raw commentary is all the data you have. You can make more games but those can be fed to both stockfish and the LLM competitor. So it seems like a valid interpretation of the bitter lesson to me.

Yeah, "hand-crafted" is a bit of a stretch; I mean their value is in the human insight they contain.

The key point I was trying to get at is that the human insights don't contain anything that can't be mined from vast amounts of gameplay. Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all.

Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.

> Every human insight can eventually be rediscovered and made rigorous by data (in chess, at least!) In the short term, those insights are useful, but in the longer term, they add nothing at all.

But isn't that the bulk of what we're shoving into LLMs, and it makes them much smarter? If it's useful there but not in a chess AI then that seems like a significant crack in the bitter lesson.

> Note also that "raw gameplay" here can mean new games -- you can generate as much data as you need, you don't need to rely on real recorded games.

Yeah I mentioned that, generated games are useful. But if we're being fair and letting both AIs use generated game data, does the more general LLM ever actually overtake the specialized stockfish like the bitter lesson suggests?

Another way to look at this is that giving the LLM the commentary is a way to avoid complaints of hiding data from the LLM, since it'll have strictly more info than stockfish. But if we cut that from the training data and only give it a basic description of chess and lots of raw game data then it's going to get even worse than it already is at chess. Meanwhile stockfish never had that commentary, just actual hand-crafted heuristics and training on game data, and it's very strong.

It's the exact opposite.

The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.

Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.

Computer chess is the canonical example of this.

But the harness still matters.

In the case of stockfish, the harness is a tree search around the neural network evaluations.

Denser sampling only seems useful if the problem domain is in some way smooth - interpolatable. If you run it on a fractal problem domain you just learn more special cases. Chess is fractal.

Maybe a future frontier LLM could approach the problem by first building its own stockfish, then applying the subsequent results

Or maybe an LLM could just tool call stockfish and doesn’t need to have more than a basic understanding of chess. The bitter lesson seems extraordinarily wasteful on the compute side.

Maybe a future LLM after that could approach the problem by first simulating a human brain, then learning from the ‘human’ gameplay.

Just kidding of course

Chess is a brute force search problem. Humans are not good at chess, even a small computer can beat Magnus Carlsen.

It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.

Both you and the parent commenter seem to be misunderstanding the point the Bitter Lesson paper makes.

The Bitter Lesson is about general-purpose algorithms vs. specialized algorithms. Historically, chess engines were programmed to look at a chess position and use positional understanding (imparted by the human programmers) to decide what the best move is. But eventually, the chess engines that actually became stronger than humans were instead programmed to just check every possible move and countermove and see which ones lead to a win. (I'm oversimplifying, but you get the point.) So even before Stockfish contained a neural network, it was considered an example of the success of the Bitter Lesson.

As it applies to AI agents, the Bitter Lesson would predict that the best possible agent would simply possess A) a way to do anything it wants, B) a way to evaluate whether what it did was correct, and C) a ton of compute. Then just turn it loose on your task. (The fact that the "brain" of the agent is an LLM is kind of irrelevant - you could also imagine the brain just being a program that generates random syntactically-correct code. What the LLM achieves is that, the random generator would take millions of years whereas the LLM is much more efficient at creating plausibly-working code. This is analogous to a chess engine's pruning heuristics.)

The hard part here is B. We've seen some great agentic successes when rewriting an existing project in a new language, since the agent can just use the project's prior test suite as its evaluator. But when developing a new project, you're still figuring out the finer details of how everything is supposed to work. As the old saying goes - writing a spec that perfectly describes how a program should work, is equivalent effort to just writing the program.

But isn't that really just about giving "front end" models more access to specialized tool libraries, which include models tuned to specific tasks? Like the first model says ah, we're being asked to code something, oh and we've been provided with some example code, let me invoke a tool call to my model the recognizes many languages, that model says that we're looking at ocaml. Okay, I better pass this off to my ocaml model which will decipher the supplied code and make a plan for what we do about the user's intent. The ocaml model recognizes that there are tests in the supplied code, let's have the special testing model have a look at the testing strategy and see how that fits in with what we just implemented, etc etc.

And perhaps at the end it all gets a single pass by a god-tier model for overall sanity and congruence, but the actual work, planning, coordination, and even user interaction was done by cheaper and faster agents of much more limited capability.

It’s kinda funny that your last paragraph is basically describing why sparse files, sparse matrices, etc. are used in other contexts.

It really is absurd to ask programming questions to a model also trained about the lifecycle of a fruit fly.

Instead of building small models from scratch, we train an enormous model and use ridiculous amounts of GPU memory. In the end, the whole thing is shoved into RAM because we don’t know where the useful parts are…

We certainly would know where they were if they were just in smaller models in the first place!

You're making a conceptual mistake here, comparing a chess tool to its operator. Deterministic tools produce superior results compared to models in many areas, so we allow models to use tooling.

The correct analogy here is Fable as a second tier player assisting a SuperGM in running stockfish, then assessing its output to identify promising variations.

There might be a limit somewhere that prevents the bitter lesson being axiomatic - for instance where simulations for anything can be exhaustive - so that judgement isn't needed any more as an arbiter. But while there are problems sufficiently complex or large to require a breadth models don't currently have, greater scale and compute will continue to convert to better decision making, and the bitter lesson will remain true (true enough).

Yes, but an LLM will just call stockfish if it needs to play chess … sure if you arbitrarily constrain an LLM to use no tools it’ll suck at chess. But no one is using LLMs in isolation. Even consumer-grade, bone-stock ChatGPT has tools.

ChatGPT does not have stockfish as a tool it can call.

Au contraire!

https://mcpmarket.com/server/stockfish

https://til.simonwillison.net/llms/mcp-in-claude-and-chatgpt...

Yeah but it can just install it. It writes arbitrary code. It can do whatever you want it to do.

Good point.

Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.

> Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.

Needs? Customers want something that immediately answers their question/solves their problem, but that’s far away, even ignoring the “run locally on a 1U server” and that that may not be in the company’s interest.

For many companies, that support line is a cost center, not a PR mechanism.

Also “Press 1 for sales, 2 for billing...” has the big advantage that it handles all accents, speech impediments, etc.

Long term I think a solution where a user’s agent trained on their voice, running on their phone communicates with the support agent of a company is where we will end up, and support phone lines will cease to exist.

Yes, and the technology to improve the interface you described is already available to run hundreds of concurrent instances on a 1U server. The barrier to entry is getting the people who manage those systems to care enough to implement something better.

Fact. My company's largest partner is CoreWeave, and convincing leadership that we could run it ourselves on partner discounted hardware for a lot less money has gone nowhere.

Maybe you need to walk into the office with a $1000 server running a hundred instances of whatever your code does.

We got rid of all our offices, whole company is remote now.

Maybe depends on how you ask it? Directly, or let it write a chess program? I think the latter can yield way better results.

No. The bitter lesson is about capabilities. GP is talking about efficiency.

GP isn’t suggesting that focused narrow model(s) will be more capable than large model, but that many small focused models can have sufficient capability while being more optimal.

Also, the bitter lesson is just wrong. The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions (eg tools internet docs).

LLMs would not be as good without expertly written context, and expert context without LLMs aren’t as good either.

> The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions

The models are not even really trained bitter lesson-style anymore. That concept peaked during the era of pre-train scaling, back when it was thought that making a bigger and bigger GPT-3 would automatically solve all problems through prompting. In 2026, the most important part of training is post-training, which uses vast quantities of niche, hand-curated data to fit the models for specific tasks in domains like tax law.

I'd respectfully push back on the framing here.

If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim)

But that misses two dimensions: 1. The cost of acquiring that output 2. What is actually "good enough" for that specialist domain

Not every output needs to be the best to produce value.

And as specialist models increase in cost, their cost/value proposition goes down.

At some point, there's a threshold where cheaper, fine tuned models are "good enough" at the task and also substantially cheaper than the expert models.

That's where fine tuning helps.

Personally, I became a believer in fine tuning after fine tuning a 1B Qwen model as a second pass over my local voice transcription app, achieving excellent accuracy at ~zero token cost and waaaay lower latency than if I'd invoked my Claude subscription under the hood.

At some point, the idea of cost/benefit analysis in the software business turned into the benefit analysis. The amount of money going into the frontier LLM model game is fantastically ridiculous. Being much better than the free resources doesn’t even touch how much better they will have to be to justify the expense of creating them, let alone continually maintaining these services.

This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely, you cannot call taking on ~1 trillion in debt and purchase commitments to scale "success". OpenAI and Anthropic are underwater financially. To be precise, they're in the Mariana Trench.

Wait, you actually found a viable counter to The Bitter Lesson? Please say more

This is a misunderstanding of either the bitter lesson or what was being claimed, on multiple accounts. Firstly, the bitter lesson is merely about human expertise-tuned algorithms vs. throwing raw compute at a domain. But, notably, it is still domain-specific. No matter how much compute you throw at training an LLM, it is never going to beat a Chess engine at Chess. If you give a Chess engine 1,000,000 compute units and a general-purpose LLM 1,000,000 compute units, the Chess engine is obviously superior at Chess; ergo, there is value in throwing compute units into training models for specific tasks. This is true for within several orders of magnitude of compute, in fact. It's also true that if you give the Chess engine 1000 compute units it'll still beat the all-purpose model with 1,000,000 units, so actually there's a lot of value in training for specific tasks.

Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute.

I don't know much about chess engines; do they still use hand-tuned algorithms, or are they more like AlphaZero, where they learn through self-play to beat any/all possible human contenders? I don't believe DeepBlue was automated to that extent, but it may have been.

In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it.

I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess.

In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction.

DeepBlue beat Kasparov with essentially raw compute thrown at alpha-beta search. That does support the premise of the bitter lesson in general. But that does not mean the bitter lesson is correctly being applied here. The point is that even if throwing raw compute at a task is better than careful human-crafted algorithms, it's still task-dependent. The current trend with the people blowing hundreds of billions of dollars is developing an all-purpose model that is everything to everyone, but you don't need hundreds of billions of dollars to create a task-specific model that outperforms their model at a given task.

> I would also be VERY slow to claim that general-purpose models will never be competitive at chess.

This is not the claim. The claim is that for the same amount of compute, a general-purpose language model will never beat a Chess model. I'm dubious, but allow for the possibility that a language model could eventually compete at a top level against humans with enough compute. However, it will never compete with a dedicated Chess model with similar resources. Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right? The bitter lesson was only about compute over human algorithms, not at throwing compute at a generalised domain over throwing compute at a specific domain.

You made arguments against two claims that I did not make (that I was trying to refute the bitter lesson or that I claimed that LLMs could never be competitive against humans at Chess), so I'd like to ask you read my statements a little more carefully this time.

> Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right?

I see arguments like this all of the time in ML. There is a large branch of statistical theory that can cast some doubt on this.

Stein's paradox: when three or more parameters are estimated simultaneously, there exist combined estimators more accurate on average (that is, having lower expected mean squared error) than any method that handles the parameters separately.

This goes deeply against "common sense", but is mathematically proven, empirically proven, etc.

If you are just taking a chess model vs. an LLM, you are right.

But if you want a chess model, a medical diagnosis model, and customer service chat model, at a high level the intuition of Stein's paradox is that one combined model will get you better performance than three task-specific models.

Most practitioners don't get this because what amounts to graduate-level stats theory is often brushed aside, and we keep relearning it the hard way.

https://en.wikipedia.org/wiki/Stein%27s_example

I don't know anything about the last 8 years of chess engines, but yea maybe 8-10 years ago AlphaZero shit all over e.g. stockfish.

I guess you missed Leela then.

And in 2020 Stockfish 12 adding some NN evaluation. And then in 2023 Stockfish 16 entirely removing the classical position evaluation code.

https://stockfishchess.org/blog/2023/stockfish-16/

AFAIK Stockfish still remains a traditional (some-variant-of-) minimax-search engine, but now uses a microscopic neural network to evaluate the positions at the maximum search depth rather than heuristics. I expect any pure ML engine can learn to exploit any fixed heuristics, but now Stockfish is trying to balance both strategies, which still heavily ends up on the side of raw search depth.

Perhaps an analogy to Moore's law?

Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner.

What countered it: Moore's law stopped working.

Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving.

Moore's Law has nothing to do with processors getting faster. Dennard scaling stopped working but Moore just slowed somewhat, not stopped.

You are technically correct. The best kind of correct.

However, what most people think of as Moore's Law--CPU speed doubles every 18 months--broke somewhere between 90nm and 22nm.

And even the actual Moore's Law--2x the transistors every 18 months--doesn't hold for all types of chips anymore. Memory only gained 2x density over 10 years.

I think the gp found a viable counter to the ggp's version of the bitter lesson, which seems so extreme as to certainly exceptions. IE, they seem to say nothing but the latest frontier model is ultimately viable as an AI business.

The actual argument of the Bitter Lesson essay is pretty limited but people's interpretation of it has gradually drifted until it's seen as prediction that current LLM will reach AGI at a large enough scale.

The issue is that GP is misusing the bitter lesson. Yes, search + learn tends to be more effective than human rules based strategies, but that's not what's being considered here. The original claim is effectively that AGI isn't needed for most tasks and more value can be created by using search + learn to solve specific problems instead of applying general models to every problem. Then GP commented a non sequitur

Are you willing to say the same of the startup or are you still stealth?

Do you have a website?

It’s not hard to fine tune a 4B parameter model to outperform GPT-5.x or Claude on a domain-specific task. You don’t even need GRPO or whatever the flavor of the month RL technique is either. I suspect what you’re saying is that massive frontier models are the best for doing big slow tasks on human time horizons—and I emphatically agree—but keep in mind LLMs are useful for quite a bit more than just multi-turn chatbots; and in those instances, throwing infinite compute and latency budgets at a problem is generally not a great way to find the “best” solution :)

I think large-scale systems are optimal when the number of tasks is relatively bounded and costs don't matter.

When you're doing something "at scale", you want smaller models, weaker heuristics and rules written in code, as they're much cheaper, much faster, much more predictable and much easier to reason about.

If your task is "tell me which pages on the internet are most relevant to this search query", running Claude Fable to evaluate every single page would be ridiculous. You may run something like it at the end, to synthesize information from a small result set for example, as that's a "human-sized" task.

I think if you start taking cost into account for tasks which models have saturated a long time ago, the bitter lesson stops being applicable.

If your task is "write a function to sort a list of 1mn entries alphabetically", a small coding model will exactly match the performance of a frontier model (it'll 1-shot it, with 100% accuracy). So the only differentiator is cost, and the frontier model will cost orders of magnitude more than the specialised model.

For most intelligence based tasks, you don't (and never have) needed the tool which "performs best at all tasks". You need the cheapest one which performs adequately for your immediate task.

This doesn't mean the bitter lesson is incorrect. At the frontier, it's still correct. It means that it's not applicable at all to lots of tasks.

Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.

VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.

The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.

So theoretically, if you give that model the means to find information, ascertain the quality of said information, it could still reason its way to an proper answer?

Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question?

> VibeThinker 3B constitutes extraordinary evidence.. math and logical reasoning

Any similar model aimed at coding?

A >10B model for mass spawning/swarming and reporting back to a larger model

To some extent VT 3B is intended as a coding model (see https://old.reddit.com/r/LocalLLaMA/comments/1u7dzdr/scaling... ). It reportedly does well on leetcode-type problems, although I didn't check that myself.

I wouldn't use it for anything important without heavy supervision, as it's very weak outside its specialty. Not ideal for instruction-following tasks.

If anyone else finds that BitterLesson link difficult to read because it wraps full-width, you can type "javascript:" in your URL bar and paste:

    document.body.style.cssText='margin:auto;width:600px;'

(Chrome strips the "javascript:" for security if I were to include it myself.)

Or use reader mode or resize the browser window.

> It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.

Absolutely false. At least when it comes to multimodal inputs, even a simple classifier will outperform the largest LLMs who still hallucinate details or don’t describe audio and images accurately.

And there’s also the issue of cost/inference speed. Running a trillion parameter model for all tasks will be incredibly costly, require a cloud API, while a tiny CNN can be run locally or at a cost multiple orders of magnitude lower.

Someone will eventually figure out how to package it all into a single, cheap chip

That you can then write text to program and make applications with.

Honestly, I've been perfectly fine using OpenClaw and Claude 4.6 for the last six months, and am used to its flaws. In general it doesn't screw up anything major, but I also wouldn't ask it to reason about anything more than a few code files at the same time. Since I'm not interested in it replacing higher level reasoning or architectural design, it's pretty capable of just doing the modifications and features I suggest.

One thing notable is that it makes a difference when I specifically refer to line numbers and variable names when I'm asking it to make changes. Meaning, I also read what it writes and understand what it's doing, and have specific queries where I already know what I'd like to change I add.

Sounds like mainframes. But then, these never quite disappeared to the date.

it feels like if we had invented hammers, and we're still on the "make them bigger, stronger" phase, but we haven't even invented nails yet.

For large language models, isn't everything nails?

Reply on news.ycombinator.com