JSON
json.org
json.org
I "discovered" a format that easily solves this, which I call QSN (quoted string notation):
http://www.oilshell.org/release/latest/doc/qsn.html
It's just Rust string literals with single quotes instead of double quotes. Rust string literals have \x01 for arbitrary bytes like C or Python string literals, but without the legacy of say \v and octal, and with just \u{} rather than \u and \U.
I use in Oil to unambiguously parse and print filenames, display argv arrays, etc. All of these are arbitrary binary data.
GNU tools have about 10 different ways to do this, but QSN makes it consistent and precise.
I'm expanding it to QTSV, a variant of TSV, but if you like you could also make some JSON variant. Technically using single quotes rather than double would make it backward compatible with JSON.
A design goal of QSN is to be human readable like JSON. You can view it at the terminal, in an editor, in an e-mail, or in the browser. IMO that fits more naturally with Unix and the web.
So I guess I should make clear that it's not only about "binary data", but it's that + a few other constraints.
Well Ion has both text and binary encodings: https://amzn.github.io/ion-docs/docs/spec.html
The 2 different types makes it unsuitable for the use cases I'm targeting QSN at, which deal only with byte strings, which may be encoded in UTF-8 (like kernel APIs).
I don't see what Ion adds over MessagePack, CBOR, or several other such formats... but that's a different discussion :)
I get that JSON being text is easy to debug, but text is just a binary format that has viewers built into everything (utf8) if we agreed on a a more structured hierarchal binary format there would be a viewer for it everywhere.
Taking it further a text file should just be one of these fictional hierarchal binary format files with a single string node for the text, maybe with some agreed metadata nodes as well, same with most other file formats.
So to convince more people to use such a format, we'd need viewers and editors that allow you to exploit the structure of the format, that are easy, intuitively and convenient to use and that are readily available on whatever system you're working on.
Text is also more robust than binary formats: Even if your json/yaml/xml/whatever file got corrupted, you can still open it in a text editor, make sense of it and fix it manually if necessary. An equivalent binary format would need to have the same property.
If we agreed upon a binary format that encodes hierarchal nodes/key/values with various data types (raw binary, strings, numbers, dates) then hopefully there would be a similar viewer everywhere and no need for most secondary decoders (json on top of text).
It should also be faster and more compact if sending binary or numbers or dates.
Properly designed it should still be able to handle corruption, but that just has a lot to with the decoder and how it handles the corruption.
JSON Numbers cannot all be represented in an IEEE754 Double, since they're arbitrary precision sequences of unbounded size.
This bit me recently, and so I'm slightly bitter about it.
But it's easily done, just as extension to the standard. The standard lacks many things and has a lot of holes. Newer standards iterations also made things worse, not better. Bad, but still the best and easiest and most secure, if you ignore the latest nonsense.
And second is that RDBMS's are historically closed-source enterprise tooling -- the vendors themselves have little interest rocking the boat, and there's not much freedom for the community to inject a new language into the system (except as ad-hoc, wonky transpilers, or framework wrappers like ORMs)
Is there any competing standard? I dont think I miss any features in JSON.
I don’t think it’s worth the cost and would rather pass some extra keys in JSON that my parser ignores (since additive changes should never cause bugs in data contracts), or a regular key ‘comment/description’.
Can you give examples? I don't see how they pose a problem.
>
|
>+
>-
|+
|-
>+#
How do you know which to use and what does what?! Especially when you see other people doing it in their code and you don't know why.
I think they could have done it more cleanly by being more verbose. Python's """ syntax or even Ruby's <<EOL
EOL
format are cleaner.
Are you able to elaborate on this problem? I'm not going to defend the complexity of YAML but I've never ran into any issues with storing complex structures within in.
> and especially the struggles around multi-line strings or other kinds of entries in YAML.
YAML actually has pretty sophisticated handling of white space within strings. The problem isn't that whatever edge case you run into can't be done, the problem is that YAML covers so many edge cases with different parsing operators that it becomes a bit of a cryptic mess trying to remember which operate is needed when. Though in fairness, JSON was never intended to be human readable (it was meant to be machine generated and machine read) so it's not any better in the readable whitespace department.
> I don’t think it’s worth the cost and would rather pass some extra keys in JSON that my parser ignores (since additive changes should never cause bugs in data contracts), or a regular key ‘comment/description’.
A third option would be to use hash-prefixed comments (like in Bash) then run that JSON through a YAML parser since technically YAML is a superset of JSON (literally, valid JSON is also valid YAML). Though I do accept that would be an unattractive option to some because you end up with less strict format checking of your source JSON (less strict in the JSON sense).
This was exactly my point around multi-line strings. You look at a mess of >'s and |'s and it's absolutely not intuitive which one you should use if the configuration files for one of the languages you're required to use happens to use YAML. In json, there's virtually no ambiguity. Everything's either a string, a number or a bool or a struct, and they all have exactly the same shape with ... really no options to make things "easier"
As for structs and arrays, YAML doesn't really make them clear, in my opinion, due to its lack of opening and closing values.
So, if you're new to k8s and need to make a configuration change to something because the darn thing doesn't work, you're forced to learn yet another markup language when it could just be a very familiar and comparatively intuitive json blob.
For example,
options:
- key: value
foo: bar
thing: thing2
smell: apple
options:
- key: value
foo: bar
thing: thing2
smell: apple
debug: true
Oh, no? It's not... well what is it?
And then a long conversation with a coworker later, they explain, "Oh! No no no, it's this:"
item:
- key: value
foo: bar
thing: thing2
smell: apple
- key: value2
debug: true
Or, in other places in some syntax, you see a bunch of:
items:
- entry
- entry2
- entry3
item: 1
item2: 2
item3: 3
{
"everything is inside curly brackets": "keys and values can be strings",
"there's a comma at the end of everything: [
"arrays exist",
"they're also comma delimited",
1, "types don't matter"
]
}Being pragmatic, I'd say neither serialisation format is better than the other. JSON does something things better (It's easier to grok nested structures and simpler to reason about the specification) but YAML does other things better (easier to embed multiline blocks of text, handles streaming better, supports comments).
Let's not also forget that most of the stuff that people dismiss in JSON is only solved by unofficial hacks (eg jsonlines) that might be widely supported but you cannot rely upon universally. So then you have two problems: a standard that doesn't support x and multiple different implementations that don't strictly support the standard. YAML is a hell of a lot better when it comes to removing undefined behaviour in parsers -- even if that does come at a cost to the complexity of the specification.
I also wonder how much of a pain it would be to add comments to json, like a json v2
However I think it goes too far. E.g. why support single quoted strings? That just makes parsing harder.
I prefer the format Microsoft uses in VSCode and Typescript - JSONC. It's just JSON but with trailing commas and comments. The downside is it isn't obvious when something is JSON and when it is JSONC because they use the same extension.
And something that a lot of language designs ignore: it makes writing harder and unnecessarily contentious. People will use them inconsistently, which has the usual effects any inconsistency has on cognitive load: causes others to question why/how/where.
In fact they're so easy they were already in JSON, and then later removed.
> I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. ~Douglas Crockford
For certain tasks (personal use) I hand roll my own json document storage database. I make a bunch of rest API calls to collect data and store it in flat files. While doing this, I use JQ to store subsets of that information (the stuff I really care about) in smaller json blobs. I then write a script to aggregate all of the smaller blobs into a larger json array.
That's almost no work and you already get a nice schema-less database. Write some commands (stored-procedures) to do any kind of filtering/modifications and you can immediately get whatever view you want of your data with one command. Write a wrapper script to identify documents by a field (primary key) and you can make data modifications in an ergonomic way. I run "modify-document <primary-key>" and it runs a tiny readline bash script prompting me for info which immediately modifies the corresponding database row.
A work flow could be...
1. Make API requests (with gnu parallel) storing the response json and a smaller json blob.
2. Aggregate the small blobs into an array.
3. Filter blobs for any that are missing information.
4. Manually update blobs missing information with readline script.
5. Filter for blobs that need processing.
6. Process blobs, use readline script to mark them as processed.
7. Continue until all blobs are processed.
This is the kind of thing I would use excel to do in the past. Hopefully I never make that mistake again.