Hacker News Reader

Comment by lacoolj | Hacker News Reader

Many tiny, specialized models is the way to go, and if that's what they're doing then it's a good thing.

Not at all, you will simply rediscover the bitter lesson [1] from your new composition of models.

[1] https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

The bitter lesson doesn't say that you can't split your solution into multiple models. It says that learning from more data via scaled compute will outperform humans injecting their own assumptions about the task into models.

A broad generalization like "there are two systems of thinking: fast, and slow" doesn't necessarily fall into this category. The transformer itself (plus the choice of positional encoding etc.) contains inductive biases about modeling sequences. The router is presumably still learned with a fairly generic architecture.

> It says that learning from more data via scaled compute will outperform humans injecting their own assumptions about the task into models.

You are making assumptions about how to break the tasks into sub models.

Sure, all of machine learning involves making assumptions. The bitter lesson in a practical sense is about minimizing these assumptions, particularly those that pertain to human knowledge about how to perform a specific task.

I don't agree with your interpretation of the lesson if you say it means to make no assumptions. You can try to model language with just a massive fully connected network to be maximally flexible, and you'll find that you fail. The art of applying the lesson is separating your assumptions that come from "expert knowledge" about the task from assumptions that match the most general structure of the problem.

"Time spent thinking" is a fundamental property of any system that thinks. To separate this into two modes: low and high, is not necessarily too strong of an assumption in my opinion.

I completely agree with you regarding many specialized sub-models where the distinction is arbitrary and informed by human knowledge about particular problems.

Aren't you just moving the assumptions to an AI model and hoping it chooses the right one for the task?

To be fair, you don't really "hope" it chooses the right ones for the task if you're optimizing the correct objective function.

so many people at my work need it just switch. they just leave it on 4o. you can still set the model yourself if you want. but this will for sure improve the quality of output for my non technical workmates who are confused by model selection.

I'm a technical person, who has yet to invest the time in learning proper model selection too. This will be good for all users who don't bring AI to the forefront of their attention, and simply use it as a tool.

I say that as a VIM user who has been learning VIM commands for decades. I understand more than most how important it is to invest in one's tools. But I also understand that only so much time can be invested in sharpening the tools, when we have actual work to do with them. Using the LLMs as a fancy auto complete, but leaving the architecture up to my own NS (natural stupidity) has shown the default models to be more than adequate for my needs.

Au contraire, ANNs are precisely the decomposition of larger problems into smaller ones.

> The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation

Is it though? To me it seems like performance gains are slowing down and additional computation in AI comes mostly from insane amounts of money thrown at it.

Yes, custom hand crafted model will always outperform general statistical models when given the same compute budget. Given that we've basically saturated the power grid at this point we may have to do the unthinkable and start thinking again.

We already did this for Object/Face recognition, it works but it's not the way to go. It's the way to go only if you don't have enough compute power (and data, I suspect) for a E2E network

No, it's what you do if your model architecture is capped out on its ability to profit from further training. Hand-wrapping a bunch of sub-models stands in for models that can learn that kind of substructure directly.

Reply on news.ycombinator.com