Hacker News Reader

Comment by pkulak | Hacker News Reader

I agree that a streaming response is cool, but why the dismissal of returning valid JSON, streamed? Why invent a new protocol when JSON already exists? Streaming JSON parsers aren't unicorns, they are horses (sorry).

With this new-line-delimited JSON format all your clients HAVE to know about your new protocol. They have to stream the response bytes, split on new lines, unescape new lines in the payload (how are we doing that, btw?), etc. If a client doesn't care about streaming, it can't just sit on the response and parse it when it's done coming in. Or, how about if later on you upgrade the system so that the response is instant and streaming is no longer necessary? Then you move on to a new API and have to keep supporting this old streaming-but-not-really endpoint forever.

View on HN

I've been using jsonlines for logfiles from about 5 years before it acquired a name.

With line delimited logfile objects, it's easy to grep for a string of interest and then only parse the lines that match -- much more efficient than parsing an entire logfile to pick out 0.01% of the lines.

That's not the usecase talked about in this article, but it is a usecase that's important to me.

Jsonlines is also an excellent intermediary format for working with any huge datasets that are just too big to be loaded into the memory all at once, and where data is hierarchical so csv is not an option. Compared to XML you save a lot in the total file size, and it's just as easy to parse.

It's also easy to load them into Google's BigQuery: https://cloud.google.com/bigquery/docs/loading-data#supporte...

I was just about to name that as an example. Newline separated JSON is a so much simpler solution to the problem than streaming JSON parsers. They're easier to understand, and it's trivial to implement.

From a purity point of view, sure, streaming JSON is more "correct", but I wouldn't call it the better solution.

http://jsonlines.org/

> Each Line is a Valid JSON Value

This seems like the better option I agree.

Google's lead me to this one today, lol: http://ndjson.org/

I'd make fun of NIH but I'm so guilty of it too.

I've just googled what jsonlines was and it turns out i've been using it for a while in log files too where I wanted JSON objects but needed it to behave like random access memory (exactly like you described with grep). I hadn't even realised it was a formal thing. Thank you

> That's not the usecase talked about in this article, but it is a usecase that's important to me.

Yeah, and I agree with you 100%. But lot's of things that are great for log storage aren't appropriate for an API.

Together with jq, jsonlines make for great logging.

it should be trivial to write a tool to stream a json file with proper newlines

Not sure what you're getting at -- my tools that generate json logfiles do trivially add newlines where I want them.

jq's "compact" mode will reformat files that don't have newlines where I want them.

I mean that the tools to add the newlines to a compact json file with minimal whitespace are not going to be the bottle neck. I am fairly sure that grep uses a forward streaming access pattern and not random access so inserting a relatively simple command into the pipeline to insert newlines, like jq, probably isn't going to slow it down.

The tool itself wouldn't be difficult to write, or like you said, just use jq

Yeah, that’s really weird. They mention streaming parsers, then immediately say “ignore them.” Why? They apparently think this solution is better somehow, but it would be nice if they’d explain why.

Also, if you’re going to reinvent the wheel and make a custom framing format, why would you choose a delimeter that can legally appear in your content? Separating your JSON with newlines is complete madness. If you’re sending UTF-8 then you can trivially use a byte that cannot appear in the data, like 0xff, as the divider.

> it would be nice if they’d explain why.

Just to fill in the picture here: it's because the built-in parser has a very good chance of being faster than anything you could write[1]. In addition, it's code that you don't own and don't have to maintain.

[1]: with exception to https://news.ycombinator.com/item?id=16413917

You’d use an existing streaming parser, not write your own.

support for un-marshaling json into objects in a streaming way is spotty / inconvenient. it should be easier than it is.

Newlines can't appear in JSON strings (they must be encoded as \n).

But newlines can appear in JSON anywhere whitespace is allowed otherwise.

Yes, you can say “we will avoid this by never generating JSON with newlines.” But why would you do that when you can easily pick a delimeter that’s guaranteed by the spec not to appear?

I’ve often used the jsonlines format for application logging, though I didn’t have a name for it until today.

I appreciate this format because, as a log file, I can use all of my standard tools to work with it (grep awk etc.). Since it’s a text file, I can use standard text editors to view and search the contents.

Since my application is the one that’s creating the log file, newlines will not appear or will be escaped.

I can often accomplish my goal without parsing the objects, but when I need to, I can break out powerful tools like `jq` or RecordStream to query the contents in a structured way. I can also treat the log file as a streaming or archived data source and process it using database technologies like Redshift or Elasticsearch.

I can’t think of another format that would be more convenient. I have worked with log file formats that are actual XML or JSON documents, and they are inconvenient in practice. I can’t think of another format that would be more convenient than this one for structured application logging.

Lots of other tools work on a line by line basis.

But newlines can't appear in JSON-Lines which is what this is really based off of.

I made this choice because so many tools support newline-delimited text. A quick search showed that others (Twitter, eBay) made the same choice, so I went with the flow.

*Thank you for NSBlog, it's great!

You’re welcome!

I think the tooling issue is for debugging and inspection, in which case you can just add a quick `tr` invocation to your pipeline to get it into a form they’ll understand.

>I agree that a streaming response is cool, but why the dismissal of returning valid JSON, streamed?

JSON hasn't been designed for chunked interpretation. How would the client know when to start interpreting the received message? How could you tell the difference between a valid chunk or malformed response?

>With this new-line-delimited JSON format all your clients HAVE to know about your new protocol

Yes. It's probably not a good option for public. It should be a complement to a more general API. I would put this in the bucket of micro-optimization for very specific use-cases. Regardless, if you publish your API and document it, other developers should be able to consume it just fine. It's not rocket science.

It works for Twitter: https://developer.twitter.com/en/docs/tutorials/consuming-st...

> carriage-return (\r\n) delimited JSON-endocded activities

That's not JSON

Why would you say that?

> Insignificant whitespace is allowed before or after any of the six structural characters.

Where whitespace may be space, carriage return, line feed, or horizontal tab. It's explicitly valid JSON.

https://tools.ietf.org/html/rfc7159#section-2

ok - but my point is that it's a protocol, that is more specific than json i.e. streaming JSON in full generality has problems. if you want to restrict yourself to a subset and create a protocol, thats great - but it's not really "streaming JSON"

> JSON hasn't been designed for chunked interpretation. How would the client know when to start interpreting the received message?

Neither has XML and yet we have plenty of working streaming XML parsers.

> How could you tell the difference between a valid chunk or malformed response?

Same as usual, when it breaks.

Streaming JSON is troublesome because the base object is unbounded. It's true that there are workhorses that handle this, but when debugging responses, ad hoc tooling is very useful. If you can't examine a response without using a streaming parser, the cost of maintenance goes up significantly.

It's important to remember why we use JSON. It's not because it's well suited to transmission. It's because it's easy to reason about. If you want to move to a streaming format that is not easy to reason about, you may as well move to a binary format.

curl is a streaming parser. Most tools know what to do with a newline.

I haven't checked, but does Chrome preview tab handle this non-standard format? That's the level where a great deal of JSON debugging occurs.

Last time I tried streaming parsing it was with Oboe.js which wasn't bad but I remember it having approximately a 5x speed penalty. So you get your first item in your dataset faster, but the whole thing loads slower - a tradeoff that's not a no-brainer. I wonder how getline plus many repeated calls to json.parse work in comparison to that - I suspect the native parser probably has some startup overhead but still better than a JS parser.

The other thing was that to get the full benefits of something like this (not necessarily about streaming parsers vs not) you had to rearrange the way your /whole/ stack works, streaming all the way from the database through all the backend layers to the frontend. It's satisfying when it works, but definitely a non-trivial amount of change.

It is a pragmatic choice. Most languages can consume newline-delimited text natively. Do you really think getline() is going away?

His point isn't that consuming data line by line is hard.

JS (and JSON by extension) does not have multi-line strings, so line endings are already escaped. All you need to do is output ‘non-pretty’ json.

Nice catch, thanks! I didn't realize that. Looks like no control characters are allowed in JSON string literals.

Let me preface by saying that I don't have an immediate solution for this case, the details seem to be application specific. However, I will note that JSON is brutally inefficient by almost every computational metric that exists and it therefore is quite expensive unless your problem domain is trivial. I used to be in the position of spending -- literally -- millions of dollars per year on JSON because that was how people wanted to format the data. You can solve many engineering efficiency problems for millions of dollars/euros.

Which is to say I don't have an answer for why streaming JSON isn't valid in this case, but I can also say that if it was up to me I would never us it in an application that mattered. It is much too expensive for many (most?) applications.

This makes more sense when working with files (rather than an API). You can use traditional Unix commands.

Another reason might be that when working with hand-written files, the delimiter adds a bit of redundancy. This make errors like unbalanced parentheses easier to diagnose.

I agree. Keeping this backward compatible would not only simplify client design but also accelerate the adoption. I am not really sure how this would be implemented, though.

It's impossible to stream a JSON Array

Embedding newlines..... hmmmmm \nI’m not sure how one would do that

Reply on news.ycombinator.com