Ask HN: How do you track NLP models in prod?

1 pointNevin19015 comments

Working on a service which uses GPT-3.5 and was wondering how I would go about tracking the usefulness of it. Wondering what you guys have done, or if you guys even track your NLP models at all.

View on HN

Are you doing classification, regression, extraction or something different? How did you evaluate the model before it went into production?

Causal text generation. I evaluated it on my own internal dataset but real world data shifts over time.

How do you get a number for text generation?

Exactly my question. I just scanned over the inputs and outputs and compared how "useful" they were.

Treat it like somebody grading an essay. It is either pass/fail or you can grade from 0 to 100 though you probably want to develop some kind of scoring rules for the latter.

I know a lot about evaluating classification but I am seeing people struggle w( evaluating text genersators; why don’t you look up my profile, send me an email, and we can talk more.

Reply on news.ycombinator.com