Has LLM killed traditional NLP?

You are not pushing it at 100. I can classify "Is 20 bottles of ferric chloride' a service or product in probably 2 seconds with a 4090. Something that most people don't realize is you can run multiple inference. So with something like a 4090, some solid few shots, and instead of having it classify one example at a time, you can do 5. We can probably run 100 parallel inference at 5 at a time. For about a rate of 250 a second on a 4090. So in 11 hours I'll be done. I'm going with a 7-8B model too. Some of the 1.5-3B models are great and will even run faster. Take a competent developer who knows python and how to use an OpenAI compatible API, they can put this together in 10-15 minutes, with no data science/scikit learn or other NLP toolchain experience.

So for personal, medium or even large workloads, I think it has killed it. It needs to be extremely large. If you are classifying or segmenting comments on a social media platform were you need to deal with billions a day, then LLM would be a very inefficient approach, but for 90+% of use cases. I think it wins.

I'm assuming you are going to run it locally because everyone is paranoid about their data. It's even cheaper if you use a cloud API.

I think your intuition on this might be lagging a fair bit behind the current state of LLMs.

System message: answer with just "service" or "product"

User message (variable): 20 bottles of ferric chloride

Response: product

Model: OpenAI GPT-4o-mini

$0.075/1Mt batch input * 27 input tokens * 10M jobs = $20.25

$0.300/1Mt batch output * 1 output token * 10M jobs = $3.00

It's a sub-$25 job.

You'd need to be doing 20 times that volume every single day to even start to justify hiring an NLP engineer instead.

That’s the argument the article makes but the reasoning is a little questionable on a few fronts:

- It uses f16 for the data format whereas quantization can reduce the memory burden without a meaningful drop in accuracy, especially as compared with traditional NLP techniques.

- The quality of LLMs typically outperform OpenCV + NER.

- You can choose to replace just part of the pipeline instead of using the LLM for everything (e.g. using text-only 3B or 1B models to replace the NER model while keeping OpenCV)

- The (LLM compute / quality) / watt is constantly decreasing. Meaning even if it’s too expensive today, the system you’ve spent time building, tuning and maintaining today is quickly becoming obsolete.

- Talking with new grads in NLP programs, all the focus is basically on LLMs.

- The capability + quality out of models / size of model keeps increasing. That means your existing RAM & performance budget keeps absorbing problems that seemed previously out of reach

Now of course traditional techniques are valuable because they can be an important tool in bringing down costs (fixed function accelerator vs general purpose compute), but it’s going to become more niche and specialized with most tasks transitioning to LLMs I think.

The “bitter lesson” paper is really relevant to these kinds of discussions.

That’s sort of like asking a horse and buggy driver whether automobiles are going to put them out of business.

I think for the most part, casual nlp is dead because of LLMs. And LLM costs are going to plummet soon, so large scale nlp that you’re talking about is probably dead within 5 years or less. The fact that you can replace programmers with prompts is huge in my opinion so no one needs to learn an nlm API anymore, just stuff it into a prompt. Once costs to power LLMs decrease to meet the cost of programmers it’s game over.

What NLP approaches are you using to solve the "is '20 bottles of ferric chloride' a service or a product?" problem?

While I agree with both you and the article I also think it'll depend on more than just the volume of your data. We have quite a lot of documents that we classify. It's around 10-100k a month, some rather large others simple invoices. We used to have a couple of AI specialists who handled the classification with local NLP models, but when they left we had to find alternatives. For us this was the AI services in the cloud we use and the result has been a document warehouse which is both easier for the business to manage and a "pipeline" which is much cheaper than having those AI specialists on the payroll.

I imagine this wouldn't be the case if we were to do more classification projects, but we aren't. We did try to find replacements first, but it was impossible for us to attract any talent, which isn't too much of a surprise considering it's mainly maintenance. Using external consultants for that maintenance proved to be almost more expensive than having two full time employees.

I suspect any solution like that will be wholesale thrown away in a year or two. Unless the damn thing is going to make money in the next 2-3 years, we are all mostly going to write throwaway code.

Things are such an opportunity cost now days. It’s like trying to capture value out of a transient amorphous cloud, you can’t hold any of it in your hand but the phenomenon is clearly occurring.

Can you talk about the main non-LLM NLP tools you use? e.g. BERT models?

> One prompt? Fair. 10? Still ok. 100? You're pushing it. 10M - get help.

Assuming you could do 10M+ LLM calls for this task at trivial cost and time, would you do it? i.e. is the only thing keeping you away from LLM the fact they're currently too cumbersome to use?

Why not just run a local LLM for practically free? You can even trivially parallelize it with multiple instances.

I would believe that many NLP problems can be easily solved even by smaller LLM models.

I see LLMs best used as part of a more traditional NLP pipeline.

For example, an approach that does me well is clustering then using LLMs on representative docs. Tools like bertopic are great for this.

I also don't see a clear cut difference between the two in certain areas. Embeddings are critical in LLM pipelines, but for me anyway, also "old school" tools.

I think NLP as described in the article is certainly under threat, but the tools and approaches compliment LLM use well, are far more efficient, and distinguish the pros from the neophytes.

If you're using LLMs for NLP-type tasks, but don't know the NLP tools, you're missing out.

So what would you use to classify whether a document is a critique or something else in 1M documents in a non-English language?

This is a real problem I am dealing with at a library project.

Each document is between 100 to 10k tokens.

Most top (read most expensive) LLMs available in OpenRouter work great, it is the cost (and speed) that is the issue.

If I could come up with something locally runnable that would be fantastic.

Presumably BERT based classifiers would work if I had one properly trained for the language.

10M items @ 10 tokens each ("20 bottles of ferric chloride" etc) plus 10M tokens out (category) is 100M tokens in 10M tokens out.

Claude Haiku is $0.25 per 1M tokens in, $1.05 per 1M out, so cost would be ~$35.

GPT-4o mini is even cheaper at $0.15 per 1M in.

Of course if your volume justifies the hardware cost you could always run Llama locally, for the cost of the electricity used.

You can use embeddings to build classification models using various methods. Not sure what qualifies as "get help" level of cost/throughput, but certainly most providers offer large embedding APIs at much lower cost/higher throughput than their completion APIs.

For context, 10M would cost ~$27.

Say Gemini Flash 8B, allowing ~28 tokens for prompt input at $0.075/1M tokens, plus 2 output tokens at $0.30/1M. Works out to $0.0027 per classification. Or in other words, for 1 penny you could do this classification 3.7 times.

That was also my impresion. LLM can "describe" but not classify. Hallucinate but nothing precise.

Prompt caching would lower the cost, later similar tech would lower the inference cost too. You have less than 25 tokens, thats between 1-5$.

There may be some use case but I'm not convinced with the one you gave.

> LLMs are not feasible when you have a dataset of 10 million items that you need to classify relatively fast and at a reasonable cost.

What? That's simply not true.

Current embedding models are incredibly fast and cheap and will, in the vast majority of NLP tasks, get you far better results than any local set of features you can develop yourself.

I've also done this at work numerous times, and have been working on various NLP tasks for over a decade now. For all future traditional NLP tasks the first pass is going to be to get fetch LLM embeddings and stick on a fairly simple classification model.

> One prompt? Fair. 10? Still ok. 100? You're pushing it. 10M - get help.

"Prompting" is not how you use LLMs for classification tasks. Sure you can build 0-shot classifiers for some tricky tasks, but if you're doing classification for documents today and you're not starting with an embedding model you're missing some easy gains.

Can you recommend a way to classify a small number of objects? Local only and Python preferably.

So TLDR: You agree with the author, but not for the same reasons?

Thank you for sharing an actual prompt thread. So much of the LLM debate is washed in biases, and it is very helpful to share concrete examples of outputs.

That’s a great example and I understand it was intentionally simple but highlighted how LLMs need care with use. Not that this example is very related to NLP:

My prompt: `<<I want a flight from portland to cuba after easter>>`

The response: ``` { "origin": ["PDX"], "destination": ["HAV"], "date": "2025-04-01", "departure_time": null, "preferences": null } ```

Of course I meant Portland Maine (PWM), there is more than one airport option in Cuba than HAV, and it got the date wrong, since Easter is April 20 this year.

How about the the costs?

link is a 404, sadly. what did it say before?

I understand the term the same way, but I don't think "traditional NLP" died with LLMs, traditional NLP already died with deep learning 7ish years ago, with LSTMs and CNNs, the gains from such models trained end-to-end are so huge it just doesn't make sense to chop up the tasks like that.

Great example. I just tried it with a few LLMs and got horrible results. GPT-4o got a ton of them wrong, GPT-1o got them all correct AFAICT but took 1m50s to do so, and Claude 3.5 Sonnet said “Here's the text with straight quotes converted to curly quotes” but then returned the text with all the straight quotes intact.

I’m very surprised all three models didn’t nail it immediately.

I would be interested how well would even a smaller LLM model work after fine tuning. Besides the overhead of an LLM, I would assume they would do a much better job at it in the edge cases (where contextual understanding is required).

I could use some help understanding, is this a set of tools or techniques to answer questions? The name made me think it's related to create embeddings but it seems much more?

I've used embeddings to define clusters, then passed sampled documents from each cluster to an LLM to create labels for each grouping. I had pretty impressive results from this approach when creating a category/subcategory labels for a collection of texts I worked on recently.

Take two documents.

Feed one through an LLM, one word at a time, and keep track of words that experience greatly inflated probabilities of occurrence, compared to baseline English. "For" is probably going to maintain a level of likelihood close to baseline. "Engine" is not.

Do the same thing for the other one.

See how much overlap you get.

It's not naive, it's how languages work.

That grammar doesn't necessarily convey 100% of semantics is a problem of natural language. Or rather, of people being poor at communicating unambiguously.

Programming languages can also be ambiguous sometimes, but that ambiguity is resolved before execution by essentially following a priority list or throwing an error if no combination of rules fits.

They all fall into the pitfall that Martin Heidegger suggested avoiding, particularly in the introduction of his text What Is a Thing?.

> This article seems to be paywalled unfortunately.

I am no fan of Medium paywalled articles but if it helps you, here's the article on archive - https://archive.is/J53CE

Like?

If you think about chunking as "take x characters" then using LLMs is a poor idea.

But syntactic chunking also works really poorly for any serious application as you loose basically all context.

Semantic chunking, however, is a task you absolutely would use LLMs for.