Hacker News Reader

Comment by chupchap | Hacker News Reader

How is this different from the categorisation models from ML era?

These are essentially zero-shot classifiers; they don't need to be trained for a specific classification task. You could include some natural language context on the rules for classification and it should get good enough accuracy.

How is it different than just asking an LLM with structured outputs enabled? Is the primary value-add that it gives a confidence level?

Mostly, they are much faster and cheaper.

1) you can parallelize the request 2) since a structured output is still just text then every previous property of the structured output effects subsequent ones

At least a year ago they were not that accurate when compared to a good many-shot classifier that (1) has 10k training examples and (2) has the learning capacity to learn from 10k training examples (most fine tuned BERTs don't seem to.)

I'm skeptical of the quality of probability calibration for models if you aren't giving them training data. The issue is that there is a prior distribution that's unique to your specific data and the calibration is really sensitive to that.

The pitch is that it's a fully-general model, so you can skip training/tuning/selecting a particular categorisation model for each task.

That's great! So someone finally built the zero-shot model from the sales decks of 2015 =D

Specifically, it happened a few weeks ago when Typesafe released Jev; this is OpenAI's competitor to Typesafe.

This isn't really anything new it just seems like a new API but you could do the exact same thing with just a little bit of prompt engineering all the way back when GPT-3 was first released. Am I missing something?

No amount of prompt engineering will give you the true probabilities for the model producing a certain response; this is something you can only get by inspecting the internal state at inference time.

For most use cases is that actually needed though? Just having it choose between predefined responses seems like enough but I'm curious about specific use cases because I do feel like I'm missing something

This is useful for classification problems; any time you need to write software that looks at some fuzzy data and needs to make a probabilistic decision. It's far more cost-efficient and performant to use this type of model instead of an LLM.

Before now you had to train a model on your specific classification problem, now these new models don't require any specific training at all to do pretty well on novel problems.

I understand but you can already do that with an LLM, it just comes down to your prompting. It is not exactly the same, but I was asking for a specific use case where that really matters

Any use case where a confidence level is desired and you want that number to actually mean something.

Becuase it is multiple orders of magnitude slower and more expensive to do it with an LLM. Look at Typesafe's Doom demo where they run complex queries tens of times per second for around $7 an hour.

There are classes of problem where this shifts the economics from "paying a human to do this is cheaper than AI" to "the software is now cheaper than the human"

"You could do this before" "No, you couldn't, this gives you something new" "Yeah but I don't want it"

... I'm genuinely just asking a question, relax

I've asked twice now about what I'm missing and for a specific use case where you can't just do this with a regular LLM call and nobody has replied that so if you have the answer that would be great. Looking for something specific instead of just it's faster or cheaper which is definitely nice but I'm just not seeing what this opens up that was not previously possible

regular LLM calls can't provide a meaningful confidence score

You’re getting answers, you just don’t like them. It’s faster and cheaper. And if you were to ask an LLM to give you a confidence score it would just be a fabrication.

I was curious about specific use cases because at the end of the day in either way you are relying on an LLM to interpret the query so whether you use a probability threshold or prompting techniques it would be a similar result, so yes this new method is much, much nicer but doesn't open up brand new use cases from what I see

It is an absolute requirement if you want to build something that (1) uses a classifier as part of a larger system, (2) can be let off the leash with no human supervision and (3) is not an AI slop demo that will be [dead] and [flagged] in seconds after being posted to HN.

For instance if you have a predictive model for market prices that is not calibrated that's... nice. If you have a calibrated model you can add a Kelly better and you have a trading strategy that makes money. Similarly if you are classifying articles or images or other contents to make a feed you might believe that 70% or 95% or some other level of precision is "good enough" and you can set the knob and turn on the cruise control.

Does this give you the true probabilities for a certain response either? How does it work exactly? The probability of an overall positive answer isn't just the probability that the next token is "Yes"

It's much faster and cheaper (an order of magnitude).

And theoretically will give you better answers statistically as it's calibrated.

Specifically, it happened over a year ago. Jev just hyped it up.

https://laya.convaiinnovations.com/

I don't know if they invented it since I've seen other papers floating around. All about marketing that's why it's hard to root for Jev's success

Reply on news.ycombinator.com