Hacker News Reader

Comment by skeptrune | Hacker News Reader

SpaCy is criminally underrated. I expect to see it experience a new wave of growth as folks new to AI start to realize all of the language tooling they need to build more reliable "traditional" ML pipelines.

API surface is designed well and it's still actively maintained almost 10 years after it initially went public.

View on HN

Is there any use case for "traditional" NLP in the age of LLMs?

Most definitely! LLMs are amazing tools for generating synthetic datasets that can be used alongside traditional NLP to train things like decision trees with libraries like cat/xgboost.

I have a search background so learning to rank is always top of mind for me, but there other places like sentiment analysis, intent detection, and topic classification where it's great too.

Do you have any sources/links that talk about this? I'm very interested in synthetic data generation, so curious what you've tried or what works / doesn't work, especially with regards to LTR.

But for the analysis use cases you mentioned, can't you just ask an LLM to read the text and output the answer as JSON, and you're done? Is it just because running LLMs is expensive?

No, it's just slow and less accurate. Wrong tool for the job when you care a lot about understanding the reasoning and internals of what the model is caring the most about.

Some low hanging fruit: SpaCy makes an amazing chunking tool for preprocessing text for LLMs.

I used to work a lot with those pipelines, I think the truth is that LLMs (and LLM embeddings) have surpassed pretty much all traditional NLP. I guess if speed is more important than accuracy? but even then, like with small embedded LLMs they still outperform "traditional NLP" on pretty much every task probably. So it doesn't make a lot of sense to not use it nowadays.

Reply on news.ycombinator.com