Hacker News Reader

Comment by simonw | Hacker News Reader

This is such a convenient format to work with. It's especially great for streaming consumption without needing to load en entire document into memory at once - you can write a streaming parser just by consuming one line at a time and passing each line to a normal JSON parsing library.

View on HN

I've seen "slight improvements" on the format which says:

- ignore any line which is exactly "[" or "]" - ignore any trailing commas

This lets you consume your API content both as a JSONL stream, and as a JSON array

not sure if it's that useful though

ClickHouse automatically detects variations of the JSONEachRow format:

- allowing optionally enclosing the data into an array;

- allowing comma as an optional separator between objects;

- allowing JSON objects without a newline separator;

- properly escaping Unicode newlines and / characters inside string literals for JavaScript compatibility;

- allowing binary data inside string literals;

Funny enough I’ve found myself just doing it naturally in cases where I was transferring a list of records. I’d still been using the json extension though feeling a bit off about it.

Recently (this year) I was flicking through a repo and saw a jsonl extension and immediately guessed what it would contain. It all just fits nicely.

It's also great for e.g. feeding search results to a client (the list-as-you-type kind), however JSONL doesn't specify how to end a stream. You will have to come up with your own higher-level protocol that allows having an EOD record.

Handling "end of data" isn't really something that any serialization format needs to specify, so I don't think this needs to be called out for JSONL specifically. e.g. in JS if you're using a ReadableStream you can just check whether a read returns .done==true. This is true regardless of the data you're reading, JSONL or otherwise

It might be too early for my brain, but splitting on "\n" won't work if any of your json objects have newlines in the strings. Does this regex work? As I see it, the tricky one is matching a single string json object.

/(\]|\})\n|(.")\n/

https://www.debuggex.com/r/LaqAP5wmPFuRLNBO

New line characters aren't allowed in JSON strings - JSON uses \n for those.

What's the difference between a newline character in a string and \n from a regex perspective?

It would be \\n

JSON doesn't have native regex support, so you would be representing a regex as a string. /\n/ in regex would need the backslash escaped, so "/\\n/"

"\n" is really just two arbitrary characters right? An actual "newline" is just a single invisible character, but we represent it as "\n" for convenience.

So in json you are simply putting in the magic character sequence: "\n" in place of the real newline character. Not unlike how many compilers will auto translate "\n" sequences you put in strings into the actual newline character.

I believe most json encoders will automatically convert newline characters into the two character sequence, so when you put it all together it goes: character sequence "\n" typed into source file -> compiler converts newline character into actual an actual newline character -> json encoder converts newline character back into "\n" sequence.

With all this in mind, since regex is basically doing the same thing as a compiler, treating "\n" sequences as a search for a newline character, you would need to escape the backslash so it is interpreted literally, so "\\n".

This gets even weirder depending on the tool/language you are doing this in though, like if you are writing out a regex within a string, the regex engine might not need to have any special logic for "\n" since the compiler is already handling it! In which case: "\\n" is an escape for the compiler not the regex engine.

TLDR: You just need to escape it: "\\n"

yes, you are correct.

  s = "foo\nbar\nbaz"
  # 'foo\nbar\nbaz'
  len(s) # 11

  j = json.dumps(s)
  # '"foo\\nbar\\nbaz"'
  # \n is escaped as \\n
  len(j)  # 15

The JSON spec [1] explicitly forbids multiline strings. (Granted JSONL is already outside the spec, but the idea there being each single line contains valid JSON, encountering a multiline string in JSONL would if anything violate more expectations.)

[1] https://ecma-international.org/wp-content/uploads/ECMA-404_2...

Reply on news.ycombinator.com