Hacker News Reader

Comment by titzer | Hacker News Reader

It's so wild that the bar for AI performance is both absurdly high and absurdly low at the same time. To specify an output format (language or grammar) for solving a computational problem is one of the oldest exercises around. On the one hand, it's breathtakingly mundane that the model can now do the most basic of tasks: conform to an output specification. It's weird reading the kind of self-congratulating blogpost about this, like OpenAI has just discovered flint knives. On the other hand, a computer system can process natural language with extremely ambiguous, open-ended problems, compute solutions to said problems, even correct its own mistakes--and then it can format the output correctly. And then on yet another hand, it only took about 10^25 floating point operations (yeah, just ten million trillion trillion, right!?) to get this outcome.

View on HN

I dont understand your complaint at all. If you develop a new revolutionary technology called an automobile, developing steering, brakes, starter, mufflers for it is a pretty big deal even if reins, clamps, mufflers and keys are mundane and have existed for decades. Structured outputs are a pretty big step in making this magic actually usable by developers as opposed to generating impressive cat pictures or whatever has captured the public imagination.

I don't think it was an complaint, just a observation.

Yes probably. But considering non-deterministic outputs is the nature of the beast with Llms and we're (mostly) engineers here, calling any part of this mundane sounds almost more like fighting words than just observation

Extremely pedantic, but is "non-deterministic" really the right language? The same input will always produce the same output, provided you haven't intentionally configured the system to use the model non-deterministically. It seems like the right way to describe it is as a chaotic deterministic system. The same input will always produce the same output, but small shifts in the input or weights can result in dramatic and difficult to predict changes in outputs.

Llms are indeed non deterministic

> The same input will always produce the same output

Not guaranteed even with the same seed. If you don't perform all operations in exactly the same order, even a simple float32 sum, if batched differently, will result in different final value. This depends on the load factor and how resources are allocated.

Yeah, the fact that floating point multiplication isn't associative is a real pain for producing deterministic outputs - especially when you're running massively parallel computations on GPUs (or multiple GPUs) making the order of operations even less predictable.

This doesn’t mean LLMs are inherently non-deterministic, just that current common implementations are non-deterministic.

Structured outputs are hard... but they claimed to have solved this a year ago.

They were lying, of course, and meanwhile charged output tokens for malformed JSON.

Structured output is trivial: just select the output tokens from the given list of probability values filtered for the allowed next tokens in the schema.

Other LLM vendors figured this out many months ago.

they did not . anthropic often sends broken json

> On the one hand, it's breathtakingly mundane that the model can now do the most basic of tasks: conform to an output specification.

I highly doubt it's the model that does this... It's very likely code injected into the token picker. You could put this into any model all the way down to gpt-2.

I wonder if you get 90% of the way with prompt engineering, and then the last 10% is just brute force, validate output, if it fails, rerun the prompt.

My assumption is if that's all this is they would have done it a long time ago though.

You can just mask the output probabilities for each token based on which options are valid according to a grammar.

There are quite a few open source implementations of this e.g. https://github.com/outlines-dev/outlines

You could simply censor invalid tokens, but that does rely on 2 assumptions.

1. There is always a valid next token.

2. This greedy algorithm doesn't result in a qualitatively different distribution from a rejection sampling algorithm.

The latter isn't too obvious, and may in fact be (very) false. Look up maze generation algorithms if you want some feeling for the effects this could have.

If you just want a quick argument, consider what happens if picking the most likely token would increase the chance of an invalid token further down the line to nearly 100%. By the time your token-picking algorithm has any effect it would be too late to fix it.

Sorry, how could there not be a valid next token? Presumably your interface would generate a state machine with appropriate masking arrays, and iirc generally speaking all 256 byte choices are in the token list. There's no way to get stuck in a place where the JSON is invalid? Can you give an example?

If you want to be really clever about your picker, a deterministic result would blat out the all the known possible strings.

For example, if you had an object with defined a defined set of properties, you could just go ahead and not bother generating tokens for all the properties and just tokenize, E.G. `{"foo":"` (6-ish tokens) without even passing through the LLM. As soon as an unescaped `"` arrives, you know the continuation must be `,"bar":"`, for example

> This greedy algorithm doesn't result in a qualitatively different distribution from a rejection sampling algorithm.

It absolutely will. But so will adding an extra newline in your prompt, for example. That sort of thing is part and parcel of how llms work

Hmm, I think any example where it can get stuck is going to be a bit contrived since really it's a question of how easy it is to recognize a valid prefix. Say for example you want the LLM to generate a valid chess match and it ends up in a situation with just 2 kings left. If you're not careful with your definitions you could end up in an endless loop that never ends.

That said if you know all valid prefixes in your language in advance then you can always realise when a token leaves no valid continuations.

> It absolutely will. But so will adding an extra newline

A newline is less likely to dramatically drop the quality, a greedy method could easily end driving itself into a dead end (if not grammatically then semantically).

Say you want it to give a weather prediction consisting of a description followed by a tag 'sunny' or 'cloudy' and your model is on its way to generate

    { 
      desc: "Strong winds followed by heavy rainfall.", 
      tag: "stormy" 
    }

If it ever gets to the 's' in stormy it will be forced to pick 'sunny', even if that makes no sense in context.

Schema needs to be a part of the prompt as well so it can associatively recall the options

You just sample from a grammar and you automatically get 100%; who knows but it seems the most likely thing they are doing. llama.cpp has supported this for a while ( using a BNF-style grammar -- https://github.com/ggerganov/llama.cpp/blob/master/grammars/... )

edit: oh actually, we do sort of know -- they call out jsonformer as an inspiration in the acknowledgements

https://github.com/1rgs/jsonformer

Oh, thanks for the links. Super interesting!

Using this in a naive way can easily degenerate into the LLM outputting syntactically/gramatically valid tokens that make no sense, like in this example: https://community.openai.com/t/json-format-causes-infinite-n...

This might be even more pronounced when the output is restricted more using the JSON schema.

So the heavy lifting here was most likely to align the model to avoid/minimize such outcomes, not in tweaking the token sampler.

Isn't your example showing an issue w/ the opposite approach, where someone is getting bad output w/ an earlier openAI json mode that worked via training rather than mechanical output restriction to conform to a schema?

FWIW (not too much!) I have used llama.cpp grammars to restrict to specific formats (not particular json, but an expected format), fine-tuned phi2 models, and I didn't hit any issues like this.

I am not intuitively seeing why restricting sampling to tokens matching a schema would cause the LLM to converge on valid tokens that make no sense...

Are there examples of this happening w/ people using e.g. jsonformer?

You're basically taking the model "off policy" when you bias the decoder, which can definitely make weird things happen.

Yeah but that's hugely wasteful of tokens.

This is like saying “we shouldn’t be celebrating a computer that can talk, my parrot can do that!”

I don't know, it doesn't sound wild at all to me. Human languages are very imprecise, vague and error-tolerant, which is the opposite of an output format like JSON. So the a model can't do these two things well at the same time is quite an intuitive conclusion.

The wild part is that a model trained with so much human language text can still outputs mostly compilable code.

I have struggled writing valid YAML before (my tokenizer doesn't handle whitespace very well). And it probably takes me a quadrillion operations on the reals to get a minimal YAML file (I think your 10^25 fp ops is an overestimate--I think it's more like 10^18-10^19).

It's kind of like an inverse Moravec's paradox.

Relatable!!

I think it will take a long time for the world at large to realize and then operationalize the potential of this "mundane" technology. It is revolutionary, and also sitting in plain sight. Such a huge technological shift that was considered decades out only a few years ago

Although I am an optimist* about what this can do, I am very much aware — from personal experience — how easy it is to see more than is really there.

The realisation of the tech might be fantastic new things… or it might be that people like me are Clever Hans-ing the models.

* that may be the wrong word; "strong capabilities" is what I think is present, those can be used for ill effects which is pessimistic.

For some reason it reminds me of my civilization runs - rush to certain high level tech and then after that discovery writing :D

It’s doing more, it is allowing user to input using natural language and the output is the json format the API that is defined

I am so often surprised by "The AI Communities" software. Often unpleasantly surprised, often just eye-rolling.

When we first started using the OpenAI API's, the first thing I reached for was some way "to be certain that the response is properly formatted". There wasn't one. A common solution was (is?) "just run the query again, untill you can parse the JSON". Really? After decades of software engineering, we still go for the "have you tried turning it off and on again" on all levels.

Then I reached for common, popular tools: everyone's using them, they ought to be good, right? But many of these tools, from langchain's to dify to promptflow are a mess (Edit: to alter the tone: I'm honestly impressed by the breadth and depth of these tools. I'm just suprised about the stability - lack thereof, of them). Nearly all of them suffer from always-outdated-documentation. Several will break right after installing it, due to today's ecosystem updates that haven't been incorporated entirely yet. Understandably: they operate in an ecosystem that changes by the hour. But after decades of software engineering, I want stuff that's stable, predictable, documented. If that means I'm running LLM models from a year ago: fine. At least it'll work. Sure, this constant-state-of-brokeness is fine for a PoC, a demo, or some early stage startup. But terrible for something that I want to ensure to still work in 12 months, or 4 years even without a weekly afternoon of upgrade-all-dependencies-and-hope-it-works-the-update-my-business-logic-code-to-match-the-now-deprecated-apis.

Well the simple answer is don't use it then.

In the same way people revert to older stable releases. You're welcome to revert to writing boilerplate code yourself.

The reason people are excited and use it, is because they show promise, it already offers significant benefits even if it isn't "stable".

"Just write your own if you aren't happy" doesn't make my critique invalid.

My problem is that this community, and thus the ecosystem, is repeating many mistakes, reinventing wheels, employing known bad software-engineering practices, or lacking any software-engineering practices at all. It is, in short, not learning from decades of work.

It's not standing on shoulders of giants, it's cludging together wobbly ladders to race to the top. Even if "getting to the top first" is the primary goal, it's not the fastest way. But certainly not the most durable way.

(To be clear: I have seen the JS (node) community doing the exact same, racing fast towards cliffs and walls, realizing this, throwing it all out, racing to another cliff and so on. And I see many areas in the Python community doing this too. Problems have been solved! For decades! Learn from these instead of repeating the entire train of failures to maybe finally solve the problems in your niche/language/platform/ecosystem)

If I wanted to be a silly pedant, I’d say that Turing machines are language specifications and thus it’s theoretically impossible for an LLM or any program to validate output formats in general.

in _general_ sure, but if you restricted each token to conform to a Kleene-star grammar you should be able to guarantee that you get something that parses according to a context-free grammar

Reply on news.ycombinator.com