Hacker News Reader

Despite reading it twice I couldn't come away with why they chose char/s or Hz as an appropriate measure. They also provided no benchmarks or model sizes except for a relative comparison with models 10x or 100x in size, which leads me to assume this is a small model maybe?

View on HN

My guess is they're generating the structure of the JSON programmatically (i.e. keys, commas, braces), and doing JSON escaping for the strings programatically, and not handling JSON in the LLM at all. Hence they're comparing char/s: first of all it's not just generating tokens, and secondly it's better for their benchmarks to compare char/s (since they don't hit the LLM for a lot of their characters) rather than LLM tokens/s (which are probably somewhat faster, but not 100x faster).

Yeh it’s weird they don’t mention parameter size or other reasoning metrics. It’s a very cool approach to getting structured output from an LLM, but the benchmarks don’t show us the whole picture. I’m wondering if their approach can be used to delegate to different models at each step in a structured output. If it could be run with mistral 8x7B and still maintain its performance then that’s awesome.

The last bit is already under way with speculative decoding but wonder what exactly are they proposing

Reply on news.ycombinator.com