Comment by cantdutchthis | Hacker News Reader

Comment by cantdutchthis | Hacker News Reader

Parent Full thread

> I think there is a good argument in saying simple NLP is basically solved

In my experience LLMs can get about 70-80% accuracy on a bunch of NER and text classification tasks if you give it a reasonable prompt. That's not nothing and it's something that you can get started with super quickly. But you'll have slow responses and typically a 3rd party running the inference.

Annotating data yourself to about 2000-3000 examples, on the datasets that I ran my benchmarks on, may get you closer to 80-90%. You'll typically also get fast inference that you can run on your own hardware no problem. By annotating the data myself I also like to think that I understand the problem much better as a consequence.

Don't get me wrong. LLMs are cool and interesting ... but they don't seem to replace old-school methods just yet.

Indeed, one of my last big projects I worked on before I retired was using an LSTM model for named entity recognition. It was many times faster than anything you can do with an LLM.

I doubt LLMs will ever replace old-school methods precisely for this reason:

> By annotating the data myself I also like to think that I understand the problem much better as a consequence.

I’m sure that once we peer into the black box, we’ll find just more refined old-school methods.

Good points. This means you can use LLMs to create an 80% accurate dataset, then manually correct the last 20% to get all the way.

"Manually" anything just depends on the size of your dataset.

Manually correcting the wrong 20% may be a reasonable amount of work, but you must examine all 100 to find the 20 that needs fixing. And that is most likely not a reasonable amount of work.

I’ve done image annotation for ML / computer vision. I found it extremely useful to use my first annotations to train a poor model, then use results from that to annotate new data. You get feedback on model quality as you go, and looking at 100 images with multiple objects is way less work than annotating 100 images.

If it’s worth it? I guess that depends on the project you’re working on.

Reply on news.ycombinator.com