Documentation for the JSON Lines text file format | Hacker News Reader

Documentation for the JSON Lines text file format

jsonlines.org

93 pointsgavmor88 comments

This is such a convenient format to work with. It's especially great for streaming consumption without needing to load en entire document into memory at once - you can write a streaming parser just by consuming one line at a time and passing each line to a normal JSON parsing library.

It's also great for e.g. feeding search results to a client (the list-as-you-type kind), however JSONL doesn't specify how to end a stream. You will have to come up with your own higher-level protocol that allows having an EOD record.

Handling "end of data" isn't really something that any serialization format needs to specify, so I don't think this needs to be called out for JSONL specifically. e.g. in JS if you're using a ReadableStream you can just check whether a read returns .done==true. This is true regardless of the data you're reading, JSONL or otherwise

Funny enough I’ve found myself just doing it naturally in cases where I was transferring a list of records. I’d still been using the json extension though feeling a bit off about it.

Recently (this year) I was flicking through a repo and saw a jsonl extension and immediately guessed what it would contain. It all just fits nicely.

I've seen "slight improvements" on the format which says:

- ignore any line which is exactly "[" or "]" - ignore any trailing commas

This lets you consume your API content both as a JSONL stream, and as a JSON array

not sure if it's that useful though

ClickHouse automatically detects variations of the JSONEachRow format:

- allowing optionally enclosing the data into an array;

- allowing comma as an optional separator between objects;

- allowing JSON objects without a newline separator;

- properly escaping Unicode newlines and / characters inside string literals for JavaScript compatibility;

- allowing binary data inside string literals;

It might be too early for my brain, but splitting on "\n" won't work if any of your json objects have newlines in the strings. Does this regex work? As I see it, the tricky one is matching a single string json object.

/(\]|\})\n|(.")\n/

https://www.debuggex.com/r/LaqAP5wmPFuRLNBO

New line characters aren't allowed in JSON strings - JSON uses \n for those.

What's the difference between a newline character in a string and \n from a regex perspective?

It would be \\n

JSON doesn't have native regex support, so you would be representing a regex as a string. /\n/ in regex would need the backslash escaped, so "/\\n/"

"\n" is really just two arbitrary characters right? An actual "newline" is just a single invisible character, but we represent it as "\n" for convenience.

So in json you are simply putting in the magic character sequence: "\n" in place of the real newline character. Not unlike how many compilers will auto translate "\n" sequences you put in strings into the actual newline character.

I believe most json encoders will automatically convert newline characters into the two character sequence, so when you put it all together it goes: character sequence "\n" typed into source file -> compiler converts newline character into actual an actual newline character -> json encoder converts newline character back into "\n" sequence.

With all this in mind, since regex is basically doing the same thing as a compiler, treating "\n" sequences as a search for a newline character, you would need to escape the backslash so it is interpreted literally, so "\\n".

This gets even weirder depending on the tool/language you are doing this in though, like if you are writing out a regex within a string, the regex engine might not need to have any special logic for "\n" since the compiler is already handling it! In which case: "\\n" is an escape for the compiler not the regex engine.

TLDR: You just need to escape it: "\\n"

yes, you are correct.

  s = "foo\nbar\nbaz"
  # 'foo\nbar\nbaz'
  len(s) # 11

  j = json.dumps(s)
  # '"foo\\nbar\\nbaz"'
  # \n is escaped as \\n
  len(j)  # 15

The JSON spec [1] explicitly forbids multiline strings. (Granted JSONL is already outside the spec, but the idea there being each single line contains valid JSON, encountering a multiline string in JSONL would if anything violate more expectations.)

[1] https://ecma-international.org/wp-content/uploads/ECMA-404_2...

> MIME type may be application/jsonl

Don't do this. Don't encourage people to use a value that's not registered yet. Say "x-jsonl", and you're OK.

> MIME type may be application/jsonl, but this is not yet standardized; any help writing the RFC would be greatly appreciated (see issue[0]).

[0] https://github.com/wardi/jsonlines/issues/19

What’s the worst that can happen when using unregistered MIME types? Is it just aesthetics or does it make some functional difference?

In a few years we will rediscover column-based, punctuation-free text files that can be parsed with single call to scanf(3).

Maybe I'm reading too generously, but my sense is that the format described in the article is intended to be used for encoding and transferring data to be used in some other application rather than something where you can assume constraints like "no punctuation". I could imagine using the "JSON lines" format for things like encoding a large number of records to insert for test cases or as a way of sharing a test dataset with another developer; I'm not convinced that either of those would work well without any punctuation whatsoever, and even if they could, I suspect I'd still rather pay the cost of slower importing in order to make my testing code (and any manual reading/transformation outside of my code) easy to do without having to add some additional processing step to deal with the punctuation.

The best argument against this format to me is that it's possible to just use JSON itself for this by wrapping all of the values in an array. I'm not totally convinced that the quality of life improvements of not needing to add commas along with the single pair of square braces necessitate an entirely new file format, but realistically I think I'll just end up following whatever happens to be supported by the tooling I use for a given project. If "JSON lines" became widely used overnight, I wouldn't be bothered by it, but if I never end up seeing it supported by anything, I doubt I'll miss it.

One particular downside to using an actual array instead of something like JSON lines is that you can't really stream the data.

If you have so much data that you need to stream it, you probably shouldn't be using JSON, though...

Shopify, for instance, uses this format if you request bulk downloads of records, for this exact reason.

>If you have so much data that you need to stream it, you probably shouldn't be using JSON, though...

Why not? I have a bunch of large/real-time files I'd like to read piece by piece and json is a convenient format for parsing it to something useful on the client. I understand using other, more efficient formats, but I think just wanting to stream the data isn't enough of a reason to move to them.

Well, I said _need_ to stream it. I think that's slightly different: I _want_ to stream data almost all the time, but I _need_ to stream data (e.g. due to RAM limitations) fairly infrequently.

If you have so much data (e.g. 8GB+, yes my workstation "only" has 8GB of RAM) that it _must_ be streamed, in my personal opinion you should opt for a better format than JSON. It's likely, too, that with a more specialized/space-efficient format, you may no longer need to stream the data!

You can, this has been done since xml was a thing (yes some of us are old). The api to do that is necessarily horrible, so no one sane does, though.

The problem is text editor support. People don't trust binary data and want to see it in a text editor. ASCII has four characters reserved for separation so ridiculous formats like xSV where x is an in-band character are unnecessary. It's just the text editor support that is lacking. Even Emacs can't seem to do it well.

I used those characters in the latter 90s when sending query results to a web client. There was a set of parameters with ADO to seriously the data or to a string. It was easy enough to split/map it back in the client.

Using hidden frames for jsonp-like request and callback support, that was the fun stuff. Don't miss it.

Also don't miss XML or SOAP.

And in a few years plus a nanosecond we will rediscover what a terrible, insecure, bug-prone, foot-gun api scanf(3) is, because of buffer overflow errors, lack of input validation, difficult error handling, clumsy newline handling, and format string vulnerabilities.

I said that you can use scanf(3), not that you should. You can use "s.split()" in python, if that makes you feel safer.

Then you are well on your way to rediscovering csv, and you still can't handle self-describing records with different structures, or nested sub-structures, or arbitrary length arrays, or null records, or unicode and control character escapes, or records containing the delimiter.

Has it occurred to you that maybe there's a reason people are using jsonl instead of an old existing standard like csv? Not just that they're such dummies they never heard of csv and would take a few years to rediscover it, but an actual reason?

Hey, you're the one who chose to recommend scanf, and claimed it would take a few years to rediscover it. I would generally assume people who read hacker news are smarter than that, and are fully aware of the security problems of scanf, and the weaknesses of csv, and are probably not crazy enough to write bulk streaming data parsers in C. And no, s.split() doesn't make me feel safer, or even do rudimentary type conversion like scanf.

Pro-tip: use a comma in your password to pollute the csv when your password gets stolen and published on the dark web.

.csv is great if you want to browse output in a spreadsheet.

Wobbly format? Sure. But if the data are purely internal, why not opt for convenience?

can't get over the fact that this page uses

> Tactile theme by Jason Long

and the format extension is .jsonl

Does anyone find themselves accidentally mispronouncing Jason like "JSON"? It must be as annoying as mispronouncing Aaron like "A-A-Ron".

https://www.youtube.com/watch?v=Dd7FixvoKBw

It's always been odd to me that "Jason" and "JSON" didn't end up homophonous. But as an A-A-Ron, if anyone named Jason ever quotes the Key & Peele sketch at me, I'll no longer need to think about how to retort!

Within my circle of friends, all Aarons get Key & Peele's "A-A-Ron" sketch quoted to them as often as all Daves get Cheech & Chong's "Dave's Not Here" sketch quoted to them.

Wait, isn't that exactly how you say JSON??

Jaysen- Jay-s-on

Here’s how I’d pronounce the two:

JSON: /ˈdʒeɪˌsɒn/ (ɒ like the o in dog, and secondary stress on the “son” part)

Jason: /ˈdʒeɪsən/ (ə, schwa, like the first a in bazaar)

A drawback of JSONL (and JSON) for me is that you can't have trailing commas. If the format allows for that, and ignores trailing commas at the end of the line you can stream and parse line-by-line, but you can also take any arbitrary group of lines, wrap them in [], again stripping the last comma (arghh!), and parse it with any standard JSON parser.

For example, Datadog doesn't ingest JSONL, so if you have JSONL log files you'll have to convert it to JSON first, you can't just use sendfile to upload your log in chunks, just wrapping it with [ and ] (which can be done with ordinary writes before and after sendfile.)

This is a good time to mention another text based data format based on readability: NestedText. You can hand edit and then pipe valid NestedText into Unix programs, which makes it useful for portability. It only supports the string type, but this can be desirable for some applications.

Find out more at https://nestedtext.org/en/stable/

If the data can conveniently be split in to lines, I wonder if there is an advantage to this format over a single json list e.g.

    [
      {"name":"Bob"},
      {"name":"Alice"},
      {"name":"Alison"}
    ]

You're already working with json anyway so you could just stay in that world rather than mixing new lines and json.

most JSON parser would have difficulty decoding an array with 1 million rows. For example in Python you could easily eat up all or RAM and swap and gets OOM'd

If we're having milions of line of anything, just go with an actual db, no?

How does "an actual db" help you with an event stream e.g. over a socket? Are you going to stream WAL records and ask the client to merge that into their local database?

No, this is often used for logging. In my last job, we were generating on the order of a million lines like this per hour in the log I worked with most often (many millions per hour if you count all the logs in the company). Each line of that log was between a few hundred bytes and ~100 KB. It would take a lot of work to support that in a database. That log was processed by many Spark jobs (some in streaming fashion, others in batch). "Small" summaries were added to a columnar database each night. The main table in that database contained hundreds of columns and billions of rows for more interactive analysis.

Why would I need a db for a meager million rows? Even excel handles a million rows

An actual db like BigQuery supports importing jsonl directly.

https://cloud.google.com/bigquery/docs/loading-data-cloud-st...

ClickHouse supports it for actual INSERT queries.

So instead of INSERT ... VALUES, you write INSERT ... FORMAT JSONEachRow and stream your data.

https://clickhouse.com/docs/en/sql-reference/statements/inse...

This doesn't work with streaming, e.g. JSON logs

The big benefit of JSON lines over a JSON array that you can easily append new records to your file by just adding new bytes at the end.

More generally, the benefit of json lines is that it's trivially streamable, you don't need a streaming json serializer or parser, because each record can be serialized or deserialized by a common eager non-streaming parser.

Ah yes, that is a big practical advantage, thanks for explaining.

If you have a streaming parser it spits out what it's got and it waits for more data (or an end which may never come). Mostly you would be extracting what you needed from the JSON as it went along. Wondering if jq works that way?

jq supports both modes. You can use --slurp to pull all newline-separated input into a big array before it starts to process it, otherwise it should produce a result for each complete object parsed (I think): https://jqlang.github.io/jq/manual/#invoking-jq

Edit: ah, the streaming described in your comment is also available via --stream, but the format presented to the selection-filtering-transformation stage will be a bit different.

I agree. I think the objections result from JSON having it's roots in JavaScript as an object. I built a streaming JSON parser for a microcontroller with very limited memory. It wasn't too hard to do though it probably doesn't cover edge cases. But it can be done if you change your mindset.

Roots? JSON IS JavaScript. Its roots are:

  var data = eval(jsonString);

This is exactly why it sucks in many ways, but also why it's pretty great.

well actually they are the same if you say (pseudocode)

``` const jsonLParser = (streamOfLines) => { for (line in streamOfLines) { if line.trim() === "[" || line.trim() === "]" -> skip if line.endswith(',') -> line = line.trimEnd(',') yield JSON.parse(line) } } ```

this allows your API to be consumed both by JSON parsers and JSONL parsers

This is useful because it gives a low-level data link framing by which a processor can recognize JSON objects without understanding JSON.

A processor that only understands the data link format (lines) can do useful things like merge or interleave two or more such streams, delete the n-th record given n, and such.

Though, I wouldn't call it a vista of limitless possibilities or anything of the sort. :) As soon as you have to crack the JSON to do anything interesting, the line orientation becomes more or less moot.

We always use:

<origin_id> <timestamp> <event_type> {payload_json} \n

This way you know what json parser to use for which line.

origin_id and timestamp is just something we always use with our EventStream

In the database it also has one iterator per origin_id, and within an origin one iterator per event_type.

This way you can listen to both an origin_id and a specific event_type while making sure you can batch fetch any missing data if the iterators skip a beat.

That page could use an intro that explains what it is. Even the examples page begins with something about CSV instead of just a minimal jsonlines example.

I THINK that the format is like a JSON file with an implicit top-level list and the top-level (only?) commas replaced by newlines? But I do not see anywhere that is spelled out.

When will this get a proper IETF RFC?

From the page:

> any help writing the RFC would be greatly appreciated (see issue).

I guess you can sign up if you want an RFC.

Are there any neat log viewers that allow you to filter and tail logs in jsonl format?

I have many systems that log in this format but are quite hard to visually scan during development. I typically create ad hoc filters using jq, rg or a js script.

I don't understand what this solves. It's JSON format, with an alternative delimiter and lack of a wrapping array.

I guess the file extension ensures that JSON is not pretty printed.

The point is that you don't need to read in the whole file before you can start working with the data.

This is great because it saves you memory (don't need to store the whole thing, your can store one line at a time depending on your specific issue), latency (don't need to wait for the whole file to be read and parsed, you start immediately after the first line), etc. Huge JSON files might be also too big to load into memory. You can read one line at a time, you can stream your lines, process your input across multiple computers etc.

And, as it's just json lines, you also don't need to invent a new notation, specialized tools, etc.

Can we PLEASE not screw this up in an actual standard and define a strict record terminator instead of separators?

Editors that remove or fail to add a final newline can go implement themselves.

NDJSON, which actually has a rudimentary spec, says

> Each JSON text MUST conform to the [RFC8259] standard and MUST be written to the stream followed by the newline character \n (0x0A).

https://github.com/ndjson/ndjson-spec/

But yeah, these things should be merged and formalized. Being web-adjacent, how likely is that? What we're seeing here is the usual patterns:

- the less-useful newer thing with slightly better marketing wins eyeballs even when it's technically inferior

- an "ecosystem cornerstone" is someone's pet project from a prior employer and now neglected (https://github.com/ndjson/ndjson-spec/issues/35#issuecomment...)

- web things don't get RFCs written because web people are disconnected from old school internet people

- W3C adjacent people didn't get their RDF tuple world domination and industry standards don't matter to them

- even MDN is switching to paywalls instead of being a useful resource

There is also a similar thing for CSV files encoded as JSON arrays - csvj.org

Can anyone explain to me what's the selling point of this format?

It's useful for streaming data where you want to pass small chunks of data incrementally. JSON by itself isn't streamable (you need the entire blob to decode it). But if we just pass smaller json-encoded chunks of data and separate them with a newline, we can make a streaming-compatible JSON protocol.

> It's useful for streaming data where you want to pass small chunks of data incrementally. JSON by itself isn't streamable (you need the entire blob to decode it).

What's wrong with JSON patch?

https://jsonpatch.com/

JSON Patch is incremental updates to a singular JSON document[1].

Lines-of-JSON is streaming delivery of multiple JSON documents. Most producers & consumers will only hold one item in memory at a time.

[1]: And kinda sucks at that job. Last time I benchmarked it, a gzipped lines-of-json stream that just repeated the whole document for every edit was smaller than a JSON Patch stream, and re-parsing the document was faster than applying JSON Patches.

I can't tell if this was created before or after all the Examples shown? It feels like people 'invented' this, then this website wrote down the implicit standard everyone already converged on.

Isn’t this already “supported” by vscode?

Rule #2 is "illegal". JSON is actually "JavaScript OBJECT Notation", and raw integers are not "Objects". For compatibilities sake, I'd strongly encourage that NLD-JSON (New-Line Delimited JSON) contains valid JSON OBJECTS.

Some discussion here: https://stackoverflow.com/questions/17224536/why-is-one-numb...

...I am by no means the harbinger of all things JSON, but it's worth pointing out that this "spec" without addressing the issues that can arise from encouraging "bare" values (not wrapped in []/{}) is highly suspect.

Raw integers are also valid JSON according to the spec: https://www.json.org/json-en.html

json is defined as one element

element is defines as ws (whitespace), value, ws

value is defined as one of: object, array, string, number, "true", "false", "null"

Wrapping in {} to make an object or in [] to make a list (which is also not an object) is not necessary to make a valid JSON document. This is also discussed in the Stack Overflow question linked.

JSON has always accepted "bare" values (try JSON.parse in the browser devtools). In the space, a valid json document is any of those "bare" values, or an array or object of those. All of the answers in your linked StackOverflow also mention this.

For example, the spec explainer at https://www.json.org/ (on the righ-hand), tells you that a `json` document is an `element`, which is a `value` surrounded by any or no whitespace. This `value` can be a literal, or an object or array.

For an (IMO) easier definition, you can check how JSON values are usually defined in TypeScript e.g. https://github.com/sindresorhus/type-fest/blob/main/source/b...

That's only true for older version of specifications (which do NOT include json.org, however). The current RFC 8259 simply defines `json-text` as any allowed value.

Isn't this just ndjson? https://github.com/ndjson/ndjson-spec

Yes -

> Two terms for equivalent formats of line-delimited JSON are:

> Newline delimited (NDJSON)[4] - The old name was Line delimited JSON (LDJSON).[5]

> JSON lines (JSONL)[6]

https://en.wikipedia.org/wiki/JSON_streaming

Pretty much, discussion: https://github.com/ndjson/ndjson.github.io/issues/1

Also when you're using this with GeoJSON there's https://stevage.github.io/ndgeojson/ which has an actual RFC (https://datatracker.ietf.org/doc/html/rfc8142)

Which itself is based on an earlier RFC for not-specifically-Geo-JSON, RFC 7464. Both do things a little different than the others: they use the "record separator" character at the start of each line and actually split on that separator when parsing.

The GeoJSON one pretty much seems to exist just to hang an "application/geo+json-seq" media type registration off of. Part of me wants to say this really should have been more of a "all json subtypes are also json-seq subtypes" situation but maybe that's not really feasible with the standards/registration processes.

It's strictly worse, actually.

ndjson specifies sane newline handling, since it works with terminators.

jsonlines works with separators and thus fails to detect truncated values. As a result, it can silently produce incorrect numeric values.

This is only an issue with plain numbers, however. If you'd have the number in an object or array, you'd detect the truncation just as well. Since using jsonl for a plain list of numbers is... overkill, I'd say it's not an issue.

On the other hand, requiring line terminators in the standard would inevitably lead to incompatibility issues. Most software would accept unterminated files, because text libraries do; and so some files will not be terminated. Some applications do not line-terminate files even on Linux (hello VSCode), and it would be even more problematic on Windows

VSCode has the ‘files.insertFinalNewline‘ setting to configure this.

It makes it incompatible with concatenative streaming, requires O(n^2) reparsing on every new chunk instead of just scanning for \n.

And if you have to parse values to detect end of record anyway, there’s no point in having jsonl standard at all, since you can just try to parse until the matching brace and repeat on success.

What’s the difference between terminators and separators here? The ndjson spec [0] doesn’t say anything like that, and it seems that ndjson and jsonlines are identical in what documents they accept.

[0]: https://github.com/ndjson/ndjson-spec

im guessing: all values must always end with a terminator, but a separator doesn't need to be present after the last value

i.e. a documemt without a newline at the end is valid jsonl, but invalid ndjson

A separator separates two records, where as a terminator terminates a record.

You can detect and error out if you see an unterminated record at end of transmission. With separators, the producer might not put a separator after the last record, because there's nothing to separate it from there.

There's no justification for not using terminators, it's just a bad spec. Unsurprising story: the variant with better marketing has less technical chops.

Reply on news.ycombinator.com