Hacker News Reader

I'm not sure about that. Newer models are able to output structured outputs perfectly and infact, if you combine it with changesets, you can have insanely useful applications since changesets also provide type-checking.

For example, in Elixir, we have this library: https://hexdocs.pm/instructor/

It's massively useful for any structured output related work.

View on HN

Any model can provide perfect JSON according to a schema if you discard non-conforming logits.

I imagine that validation as you go could slow things down though.

Expect this is a problem pattern that will be seen a lot with LLMs.

Do I look at whether the data format is easily output by my target LLM?

Or do I just validate clamp/discard non-conforming output?

Always using the latter seems pretty inefficient.

The technical term is constrained decoding. OpenAI has had this for almost a year now. They say it requires generating some artifacts to do efficiently, which slows down the first response but can be cached.

Reply on news.ycombinator.com