Hacker News Reader

Comment by srameshc | Hacker News Reader

As someone who has very limited understanding but tried to use BERT for classification, is BERT still relavant when compared to LLMs ? Asking because I hardly see any mention of BERTs anymore.

View on HN

Yes, they are still used

- Encoder based models have much faster inference (are auto-regressive) and are smaller. They are great for applications where speed and efficiency are key. - Most embedding models are BERT-based (see MTEB leaderboard). So widely used for retrieval. - They are also used to filter data for pre-training decoder models. The Llama 3 authors used a quality classifier (DistilRoberta) to generate quality scores for documents. Something similar is done for FineWeb Edu

Wait, I thought GPT's were autoregressive and encoder only like BERT used masked tokens? You're saying BERT is auto-regressive or am I misunderstanding?

You're right. Encoder only models like BERT aren't auto-regressive and are trained with the MLM objective. Decoder only (GPT) and encoder-decoder (T5) models are auto-regressive and are trained with the CLM and sometimes the PrefixLM objectives.

You can mask out the tokens at the end, so its technically autoregressive.

For many specialized tasks you can run BERTs (and simpler models in general) at scale, with lower latency, at lesser cost, with similar or even better results.

They're still very useful on their own. But even more broadly, you can often use them in tandem with LLMs. A good example could be a classifier that's used as a "router" of sorts; could be for selecting a prompt template, directing to a specific model, or loading a LoRA or soft prompt vector to be used at inference-time.

Depends what you’re trying to do. I’m writing a personal assistant app (speech to text) and want to classify the user input according to the current actions I support (or don’t). The flagship LLMs are pretty great at it if you include the classes in the prompt and they will spit out structured output every time. But, man, they are expensive and there’s the privacy aspect I’d prefer to adhere to. I’ve only got 24 GB of RAM, so I can’t run too many fancy local models and things like llama3.1:8b don’t classify very well.

So I’m trying BERT models out :)

Try some of the Quen models. They have some that are slightly larger than 8b that will fit on your 24gb quite nicely. They have been amazing so far.

Wanted to circle back. The 2.5 coder 7b instruct is amazing, holy crap. It classifies my cases almost perfectly and almost always returns perfectly formatted JSON.

Yeah, i've been seriously impressed. Glad it worked for your use cases.

My understanding is that BERT can still outperform LLMs for sentiment classification?

To my understanding yes. But I never found a good use-case for sentiment classification.

It seems to be used by Youtube for comment censoring / shadow-banning.

That might make sense.

I used sentiment analysis a few times in recommender systems (for digital media consumption.)

Also for analyzing Trump's tweets (from 2016): https://mathematicaforprediction.wordpress.com/2016/11/21/te...

Really cool investigation. Would you mind sharing the BERT specific sentiment detection?

Thanks!

See the Mathematica code in this Markdown file: https://github.com/antononcube/MathematicaVsR/blob/master/Pr...

Safety

They’ve drowned in the LLM noise, but they’re definitely still relevant.

- Generative model outputs are not always desirable, and often even undesirable

- BERT models are smaller and can run with lower latency and serve larger batches with lower vram requirements

- BERT models have bidirectional attention, which can improve performance in many applications

LLMs are “cheap” in the sense that they work well generically, without requiring fine tuning. Where they overlap with BERT models is mostly that they may work better in low training data environments due to better generalization capabilities.

But mostly companies like them because they don’t “require” ML engineers or data scientists on staff. For the lack of care given to evaluation that I see around LLM apps, I suspect that’s going to prove to be a faulty premise.

> - BERT models are smaller and can run with lower latency and serve larger batches with lower vram requirements

The most recent version of Wolfram Language (aka Mathematica) uses by default BERT models for embedding.

(Say, for this function: https://reference.wolfram.com/language/ref/CreateSemanticSea... .)

I think berts are still used for stuff like embeddings (at least the best generic embedding model in my language is bert based), when you want to get a model that has basic semantics but needs to be fast.

Still relevant in DNA space! https://arxiv.org/abs/2402.08777

BERT is favored for generating embeddings and bag of words strategies for clustering

Reply on news.ycombinator.com