Sonic: A fast JSON serializing and deserializing library

From the documentation it seems to me that they're mostly concerned about resource utilization when processing APIs, and not dealing with files. 635KB may be representative of a large API payload in their environment.

simdjson is listed thought.

Depends. I've had multi-gigabyte `[{..}, {..}, ...]` json arrays from database dumps, and doing even basic things with that with jq takes ages unless you use the (highly obtuse IMO) streaming methods. Sometimes you can pre-grep to filter the results to something trivial to process, but sometimes the structure is not unique enough to let you do that, or it depends on multiple field values - filtering that with a json parser makes perfect sense, and then speed can matter.

That said, a 2x+ improvement for a couple megabytes, especially if done many times per second, is still a significant improvement.

Huge log files are one use case. Large datasets are another, in data analysis, machine learning, and ETL tasks.

GeoJSON data can be just as big as you’d like. I have a big archive of GIS data where the files are enormous.

Hardware config file for a small SOC. It faithfully describes every addressable physical byte.

Not JSON, but we process XMLs on the order of a GB. Largest ones are consolidated invoices (ie a lot of separate invoices in one file). Other large ones contain rules and codelists in multiple languages.

All the time. It's not there's a single record that is that size, but all sorts of things log in json, so you wind up with multi-GB jsonl files. As an example: AWS CloudTrail logs.

Monthly general ledger entries for the largest real estate companies, tried XML and JSON, eventually landed on compressed CSV for best trade off between human readable large files (~1-3GB) and compressibility.

635kb is indeed "large" if it's one message. (You will have millions of messages in one file if you're doing batch processing.)

I thought UTF-8 was the standard encoding that the json spec requires?

somehow when I saw it's a fast JSON serializing salary library I knew it's for Go. For some reasons there're always new JSON libraries for it

web development -> REST -> JSON -> JSON parsing libraries

So like js frameworks, json parsers are everywhere

It sounds like it essentially is written in C. INTRODUCTION.md says:

> As for insufficiency in compiling optimization of go language, we decided to use C/Clang to write and compile core computational functions, and developed a set of asm2asm tools to translate the fully optimized x86 assembly into plan9 and finally load it into Golang runtime.

Github says it's 59.6% assembly and 6.5% C. Possibly the assembly is just the checked-in result of compiling+translating the C?

Gross that they have to do this. I know the Go folks really prioritize speed of compilation, but I wish they'd support debug builds with their own backend and release builds with LLVM so you could get this kind of performance when actually writing Go. I see there have been a few attempts at Go + general-purpose backend (gccgo, llgo, gollvm) but none seem to use the official Go frontend written in Go, so I think they're doomed to be second-class at best.

edit: and/or, if the Go folks and GCC and/or LLVM coulds could negotiate a shared ABI (not necessarily switching to the platform's default C ABI but having "cc --abi=special-golang-stack-copying-thing"), folks could just link against something compiled in C/Rust/whatever without requiring this separate compilation+translation step (the high-maintenance, high-performance path) or CGo overhead (the easy but slow-for-frequent-calls path).

I really don't care what language it is written in. As long as there is full test coverage, I'm happy.

I've been a long time user of jsoniter and it is much faster than the standard lib. It really makes a difference for the work I do. If this is even faster and has tests, even better.

Using Go assembly lets you avoid the Cgo overhead IIRC.

Which probably means its solving a real world and modern problem they encounter and carries the same weight as a library released by Google

At that point you might as well ignore the fact it's JSON and just treat it as a stream.

Out of curiosity, what does "working with" mean here? What operations did you need to perform on it? Streaming reads, transforms, indexed reads, appends, edits?

I'm thinking that any general-purpose JSON loader is likely to perform badly for a 100GB file, purely because it'll use 2x (or more likely 10x) as much memory for the parsed representation. So you'd want some kind of special-case reader for huge files -- maybe it just builds some kind of sparse index with pointers back into an mmap of the raw data.