Hacker News Reader

Comment by catlifeonmars | Hacker News Reader

They almost certainly would perform worse than more specialized classifiers trained with less data. It’s kind of a paradox of generalization. I think there’s an interesting space where you use generalized models to generate ad hoc specialized classifiers.

View on HN

Depends what you man by "more specialised". You wont train very good language understanding without alot of data. It probably uses the core tranformer stack from an LLM.

Classic classifiers are regularly just tuned general models; Training a CCN on ImageNet and tune it for cats and dogs gives better results than just training it on cats and dogs.

There is likley a small network used to tranform model output vector to probabilities, but that wouldn't be massive. Retraining that small network for specific task may beat jev; but that's bairly considered training by modern standards.

Expecting a strong zero-shot performer to perform worse in a low data regime?

That only makes sense if you try to rope in data previously used to establish the model's priors, but that wouldn't make sense in this context. That same additional data is what enables things like...

> use generalized models to generate ad hoc specialized classifiers.

They almost certainly would perform worse than more specialized classifiers trained with less data. It’s kind of a paradox of generalization.

Isn't this exactly what the bitter lesson is about?

Reply on news.ycombinator.com