Hacker News Reader

Comment by djha-skin | Hacker News Reader

The answer to most of these kinds of questions of generalist versus specialist when it comes to AI, according to Richard Sutton's "The Bitter Lesson"[1], is that specialized AI will win for a while, but generalized will always win the long-term because it's better at turning compute into results.

Whether or not that will pan out with "fine tuned" versus "generalized" versions of the same data-eating algorithms remains to be seen, but I suspect the bitter lesson might still apply.

1: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

View on HN

Well the counter argument is that with an equal amount of compute, specialized models will win over generalized ones. The reason is that generalized ones have to spread out their weights to be able to do a ton of irrelevant things to the niche subject that a finetuned model can accomplish with fewer and more focused weights.

I'd argue that yours is not a counter argument, but the same thing said by grandparent rephrased. It's obvious that with the same amount of computing power, specialized > generalized. But in 2-5 years, the future generalized models can beat the current specialized ones.

The future of this is the same as what happened with PCs. The specificity will initially be thrown away, added back later as an accelerator, and eventually, brought back into the fold as an automatically used accelerator. It all comes full circle as optimizations, hinting, tiering, and heuristics.

AI will be used to select the net to automatically load. Nets will be cached, branch predicted etc.

The future of AI software and hardware doesn't yet support the scale we need for this type of generalized AI processor (think CPU but call it an AIPU.)

And no, GPUs aren't an AIPU, we can't even fit whole some of the largest models on these things without running them in pieces. They don't have a higher level language yet, like C, which would compile down to more specific actions after optimizations are borne (not PTX/LLVM/Cuda/OpenCL.)

No, the GP is stating that the short term advantage will always exist, for fundamental reasons. It's just short term because it has to be reacquired often, not because there stops being an advantage.

In 5 years, the specialized model would still beat the generalized ones. They would just be different from the ones today.

Couldn't you apply the same prompt engineering tricks to the specialized model that their system beats?

Well, I'm not sure how valid is the entire thing. It's not clear how much cross-polination there is between those datasets and the ChatGPT training set (IMO, if I was training something like it, I'd take any specialized dataset I can find), or how useful answering those questions is to any real task.

Also, it's not only the prompt engineering. Training ChatCPT was a decades long process, with billions of dollars invested, and it takes a small cluster to run the thing. How would the other models compare if given similar resources?

Besides that, on the context of this thread, the bitter lesson has absolutely no relation to the specialized vs. general purpose model dichotomy. It's about the underlining algorithms, that for this article are all very similar. (And it's also not a known truth, and looking less and less as an absolute truth as deep learning advances, so take it with a huge grain of salt.)

That's probably true in terms of benchmark problems, but at the same time, losing knowledge of seemingly irrelevant things could make the specialised model less creative when it comes to looking for unexpected causes and links.

It would also be less robust in the face of exceptional situations that should make it doubt the reliability and relevance of its own training.

For instance, an autonomous driving system that doesn't know the first thing about the zombie apocalypse could put its passengers' lives at risk by refusing to run over "pedestrians".

A specialised diagnostic system might not notice signs of domestic violence that a GP would see. Being able to connect the dots beyond the confines of some specialised field is extremely useful in many situations.

> For instance, an autonomous driving system that doesn't know the first thing about the zombie apocalypse could put its passengers' lives at risk by refusing to run over "pedestrians".

I think that argument works better the other way. I'm outspoken in arguing many of the edge cases in driving are general intelligence problems we're not about to solve by optimising software over a few billion more miles in a simulator, but I don't want that intelligence so generalised that my car starts running down elderly pedestrians because there's been a lot of zombie literature on the Internet lately. I'm pretty confident there's more risk of people dying from that than a zombie apocalypse.

> For instance, an autonomous driving system that doesn't know the first thing about the zombie apocalypse could put its passengers' lives at risk by refusing to run over "pedestrians".

I'd prefer my self-driving car not to learn about the zombie apocalypse, lest it starts running over pedestrians.

Wouldn't it become more and more difficult to improve the specialist model? And wouldn't the ultimate generalized models (AGI) eventually be able to improve themselves automatically? Or am I talking about something different here?

This is an empirical observation regarding implementation details of machine learning models specifically, not math models in general. Clearly, in principle, specialized models can beat general models simply because at least some applications have exact models. No learned functional approximator will beat an ALU or even an abacus at simple arithmetic.

Well said, and fully agree. If you horse race two approaches, you can of course arbitrarily arrive at a winner based solely on the version of each approach you choose. You need a deeper look if you want to generalize.

I think it might be stretching the Bitter Lesson a bit to extend it to cover fine-tuning, since the original idea was addressing something very different.

Anyway, even if generalist models are truly better at everything, they’ll never be faster or cheaper than their specialist counterparts, and that matters.

In Sutton's essay he's talking about more general techniques vs ones that build in expert knowledge. In the case of this paper it's the same technique in both the fine-tuned and "general" cases. So Sutton's argument doesn't apply here. This is entirely about training data.

The big flaw in this paper, to me, is that no one knows what GPT-4 has been trained on. It could include all the same training sources as the specialized models for all we know. If that's the case, it's a bit curious that it requires prompting to get the accuracy, but again, we'll never know.

I'd go so far as to say this paper isn't even giving any useful information here, aside from the fact that GPT-4 can be made to work well in these specific domains with prompting.

I mean the whole point of the “bitter” part is that if you push most people that aren’t convinced of the data hypothesis, they will talk of a belief that there’s “something special” that happens to create intelligence that isn’t just raw number crunching (so to speak). It’s a human bias in how we view social hierarchy and expertise, and ignores the epistemic/biological perspective.

It’s a philosophical position as much as a technical one

no - because the range of responses in the generalized model will tend in some asymptotic way to "patterns the machinery recognizes" . Real knowledge is not like a machine process, in every instance. Any detective novel has a moment when a faint scent or a distant sighting connects very different inputs to aid in solving a mystery. Lastly, knowledge has resonance and harmonics; great Truths are paradox. Your bitter lesson is convenient at this time of evolution with computing, not The Answer.

Good read, thanks for sharing.

Reply on news.ycombinator.com