Hacker News Reader

Comment by holdenc137 | Hacker News Reader

I don't get it (and I'd call this cumulative not incremental)

Why not at least wait until the key is complete - what's the use in a partial key?

Doesn't it do exactly that?

> As a consequence of 1 and 5, we only add a property to an object once we have the entire key and enough of the value to know that value's type.

Their example in the README is extremely misleading then. It indicates your stream output is

name: A name: Al name: Ale name: Alex

Which would suggest you are getting unfinished strings out in the stream.

How is it misleading? It shows that it gives back unfinished values but finished keys.

Cumulative is a good term too. I come from the browser world where it's typically called incremental parsing, e.g. when web browsers parse and render HTML as it streams in over the wire. I was doing the same thing with JSON from LLMs.

If you're building a UI that renders output from a streaming LLM you might get back something which looks like this:

  {"role": "assistant", "text": "Here's that Python code you aske

Incomplete parsing with incomplete strings is still useful in order to render that to your end user while it's still streaming in.

incomplete strings could be fun in certain cases

{"cleanup_cmd":"rm -rf /home/foo/.tmp" }

Yeah, another fun one is string enums. Could tread "DeleteIfEmpty" as "Delete".

I imagine if you reason about incomplete strings as a sort of “unparsed data” where you might store or transport or render it raw (like a string version of printing response.data instead of response.json()), but not act on it (compare, concat, etc), it’s a reasonably safe model?

I’m imagining it in my mental model as being typed “unknown”. Anything that prevents accidental use as if it were a whole string… I imagine a more complex type with an “isComplete” flag of sorts would be more powerful but a bit of a blunderbuss.

If any part of that value actually made it, unchecked, to execution, then you have bigger problems than partial JSON keys/values.

Incremental JSON parsing is key for LLM apps, but safe progressive UIs also need to track incompleteness and per-chunk diffs. LangDiff [1] would help with that.

[1]: https://github.com/globalaiplatform/langdiff/tree/main/ts

Why not just chunk the json packets instead?

Only because you have access to the incomplete value doesn't mean you should treat it like the complete one...

In this example the value is incomplete, not the key.

Reply on news.ycombinator.com