Hacker News Reader

Comment by refulgentis | Hacker News Reader

Yes (rationale: 3.1 was, would be strange to rollback.)

In general, you'll do a ton of damage by constraining token generation to valid JSON - I've seen models as small as 800M handle JSON with that. It's ~impossible to train constraining into it with remotely the same reliability -- you have to erase a ton of conversational training that makes it say ex. "Sure! Here's the JSON you requested:"

View on HN

What kind of damage is done by constraining token generation to valid JSON?

Yeah, from my experience if you prompt something like:

respond in JSON in the following format: {"spam_score": X, "summary": "..."}

and _then_ you constrain the output to json, the quality of the output isn't affected.

What about OpenAI Structured Outputs? This seems to do exactly this.

Correct, I think so too, seemed that update must be doing exactly this. tl;dr: in the context of Llama fn calling reliability, you don't need to reach for training, in fact, you'll do it and still have the same problem.

I'm building this type of functionality on top of Llama models if you're interested: https://docs.mixlayer.com/examples/json-output

I'm writing a Flutter AI client app, integrates with llama.cpp. I used a PoC of llama.cpp running in WASM, I'm desperate to signal the app is agnostic to AI provider, but it was horrifically slow, ended up backing out to WebMLC.

What are you doing underneath, here? If thats secret sauce, I'm curious what you're seeing in tokens/sec on ex. a phone vs. MacBook M-series.

Or are you deploying on servers?

Reply on news.ycombinator.com