Comment by SigmundA | Hacker News Reader

Comment by SigmundA | Hacker News Reader

Parent Full thread

Text is a binary format that just happens to have decoders (ASCII, UTF8 etc) everywhere in order to parse and display it to humans along with more specific parsers for things like JSON.

I wish we could all settle on a binary structured format thats not limited to text encoding and more like JSON (key value) that basically everyone uses with real data types (efficiently encoded numbers, dates, binary , etc).

Text files would just be something like { contentType: "text/plain" content : "text here" } while an image could be { contentType: "image/jpeg", content: <real binary data not base64> } you could also add whatever other metadata you want and all of it is retained and easily parseable. Other more structured formats obviously wouldn't just be a blob for content and every system out there would have a structured binary format viewer/editor just like there are text viewers and editors now.

Sqlite is kinda used this way and has some nice properties like indexes and transactions but is relational instead of hierarchal which can be good and bad, not as straight forward to just view or navigate. Protobuf is another used quite a bit now, honestly I don't care just something everyone agree upon that can encode more structure efficiently but still can be easily inspected everywhere.

Doubt this will any time soon but it does seem inevitable in the long run that we figure out a way to send data between system and what a string, number, date etc is and stop having text encoding and escaping issues.

Something like UBF might be interesting to you: https://ubf.github.io/ubf/

You mean like CBOR?

https://en.wikipedia.org/wiki/CBOR

Sure, looks good, plenty to choose from, just need to get everyone to agree on one.

Although I do like ones that have some concept of a schema to reduce repeated key size and allow validation.

There's also Minecraft NBT. It's a bit obscure but quite nice. https://minecraft.fandom.com/wiki/NBT_format

You should put your NBTs inside of a netstring, https://en.wikipedia.org/wiki/Netstring

Or indeed ASN.1 [0], which started life in 1984 and is still used in a lot of places today.

The idea has been around for almost 40 years in arguably more complete format than any of the above up to and including a standardised schema representation.

It's not the lack of a suitable standard that's holding it back.

[0] https://en.wikipedia.org/wiki/ASN.1

Although it's not a format, the closest thing I have found is Postgres. You get well defined types that make life easier. They are stored efficiently in binary. And have a sensible text representation that means you can still read and write them by hand. I can store a PostGIS geometry with millions of verticies and still use text to create a point or read an envelope. Text formatting is hugely useful and benefits from standardisation. But that does not need to be built into the format.

>{ contentType: "image/jpeg", content: <real binary data not base64> }

It would have to be encoded somehow, because what if the "real binary data" included the byte 0x7D which is '}'

Sorry thats not meant to be a literal format but conceptually mapped to text JSON. It would instead be more like CBOR or other JSON like binary formats at the byte level.

I don't know why the ASCII characters for record and field separators don't get more love. That is what they are there for. My suspicion is because they aren't type-able, so people rarely encounter them.

Still have the issue of binary data encoding as well since the data will have those bytes in it so they need to be escaped vs a format that defines how to encoding binary as is efficiently (length delimited).

Reply on news.ycombinator.com