Hacker News Reader

Comment by kstenerud | Hacker News Reader

It's not just a problem with data movement inside the machine. Data movement between processes and machines is also horribly wasteful.

Text encoding/decoding wastes massive amounts of energy, which is why I've been developing a format over the past 2 years with the benefits of binary (smaller, more efficient) as well as text (human readability/editability): https://github.com/kstenerud/concise-encoding

View on HN

> Data movement between processes and machines is also horribly wasteful

I agree, and in .NET that’s a solved problem: http://const.me/articles/net-tcp/ (2010).

Their text format is XML. Their binary version takes small fraction of bandwidth, especially when using a custom pre-shared XML dictionary. Binary is much cheaper to produce or parse, the built-in DataContractSerializer understands both formats. Supported by all modern editions of .NET including .NET Core and is compatible across them, e.g. I’ve been recently using it for a network protocol of a Linux ARM device.

Waste can be very subjective.

Exchanging text makes the format easier to debug and accessible to humans, which can be a purpose in itself.

100% agreed, which is why Concise Encoding has 1:1 type-compatible binary [1] and text [2] formats, which can be automatically converted when needed with no data loss.

Human accessibility is an absolute must for any modern format.

[1] https://github.com/kstenerud/concise-encoding/blob/master/cb...

[2] https://github.com/kstenerud/concise-encoding/blob/master/ct...

It should only be easy to debug and accessible to humans when needed IMO. I really think it should be possible to load a "production message", log event or whatever into a debug parser which only then spits out human readable output.

Even switching between modes may introduce new kinds of undefined behavior, which may not be visible on the text-only side

Every extra step in the process adds another potential failure point. But we're fast approaching a data and energy crunch that will push the industry towards binary formats once more. This is my attempt to keep that shift sane, and avoid the mess of the 80s and 90s.

The implementations are almost done now, and my first tool will be a command line utility that reads one format and spits out the other, so that you can take a binary dump from your production system using tcpdump or wireshark or whatever, and then convert it to a human readable format to see what's going on. I'll probably even put in a hex-reader so that you can log the raw message and then read it back:

    2019-10-02:15:00:32: Received message [01 76 85 6e 75 6b 65 73 88 6c 61 75 6e 64 68 65 64 79]

    $ ceconv --hex 01 76 85 6e 75 6b 65 73 88 6c 61 75 6e 64 68 65 64 79
    v1
    {
        nukes = launched
    }

I doubt there is any significant crunch coming from just encoding/decoding the stream. Most of the time comes still from waiting for network resources, and evaling megabytes of add-in JS. Now the same problem applies to JS, and the counter argument for openness and usability are still the same. Compressed transmission and binary parallel transmission in HTTP/2 are also helping with the comms size.

The project still seems cool, I'll have to have a deeper look into it soon

This is pretty cool! I really like that dates (and timezone info) are first class types. Often neglected, time ser/de has been a monkeywrench in many a project.

Protobufs are neat but `protoc` is an absolute nightmare. I would warn against going down the path of transpiling to lang-specific message files if at all possible (I don't think you are taking that approach, but github freezes my phone browser, I'll look more on my dev machine)

Great stuff! I would love to see this get more traction. Interchange formats are a classic victim of critical mass necessity.

How does this compare to MessagePack? (https://github.com/msgpack/msgpack)

MessagePack:

* Doesn't have a sister text format for human editing.

* Uses raw uncompressed types for integers and floats, which wastes space

* Doesn't support arbitrary sized/precision numeric types

* Timestamp format uses seconds instead of Gregorian fields (which means that your implementation must keep a leap second database, and time conversions are complicated)

* No time zones

* Doesn't have metadata, comments, or references

* Container types have an unnecessary length field

* Stores data in big endian byte order (an unnecessary expense since all popular modern chips are little endian)

* Wastes an extra bit for negative integers

* 32-bit limitations on length (so max 4gb payload sizes)

* No versioning of documents

That's actually a pretty convincing list! You might want to consider adding a feature comparison matrix for a few of the more popular formats in a prominent location. It also probably wouldn't hurt to lead the page with a small example of encoded text, the way MessagePack does (it's what got me to give their website a closer look, which led to GitHub, and then I started using it).

Regarding the point about having a sister text format, I suppose I'd tend to view JSON as filling that role. Not to bother you for a second comparison, but if you happen to have it handy seeing how CTE compares to JSON in terms of encoding efficiency would be interesting to me.

Edit: I found a list of MessagePack <--> JSON limitations that I hadn't really thought about! (https://github.com/ludocode/msgpack-tools#differences-betwee...)

Thanks for the feedback! I'm not exactly the greatest at marketing, so any help in this regard is most appreciated :)

I've worked with lots of binary formats, and they are equally "human-readable" (and far less unambiguous in a lot of cases) once you get used to it. As a coworker once said, "everything is human-readable if you have a hex editor."

It seems your format is yet another attempt at reinventing ASN.1.

But the point is that binary formats shouldn't require looking at hex data just to make sense of them. Especially not nowadays. Once you're stuck parsing binary data by hand (which is tedious), you've lost the amazing benefit that text formats gave us. And editing binary formats is even worse.

That's why I split it into compatible text and binary formats. You transmit/store in binary, and read/edit in text, and the computer deals with the conversion automatically.

I think this is the direction things are heading. Currently, I use yaml for tons of configs and the like because it's super easy to modify. I use jq to process because jq is the bee's knees. There is a yq out there which can read most config formats, but it doesn't support the full jq syntax, so instead I pipe everything through `yq read -j -` to convert to json and then `| jq 'query'`. works a treat.

I can see with a few more small parts an application which persists as bson, streams / tempfile / and/or mmaps to yaml/hcl/whatever you like, opens that in your editor of choice, and translates back to bson.

Seems like a good effort. I couldn’t easily find an example of the text encoding looking at your readme though.

Text: https://github.com/kstenerud/concise-encoding/blob/master/ct...

Binary: https://github.com/kstenerud/concise-encoding/blob/master/cb...

Reply on news.ycombinator.com