Hacker News Reader

Hacker News

Comment by kstenerud | Hacker News Reader

Full thread

kstenerud

Plain text protocols are:

- human readable

- good for quick protyping

- good for inspection while debugging

They are also:

- complicated and slow to parse

- more bloated than binary

We benefited from text protocols because we had plenty of headroom: memory was cheap enough, storage was cheap enough, network was cheap enough, power was cheap enough for our use cases. But that's not quite so true anymore when you have to scale to support many connected systems and handle lots of data. The honeymoon's almost over.

These are some of the reasons why I'm building https://concise-encoding.org

View on HN

> We benefited from text protocols because we had plenty of headroom: memory was cheap enough, storage was cheap enough, network was cheap enough, power was cheap enough for our use cases. But that's not quite so true anymore when you have to scale to support many connected systems and handle lots of data. The honeymoon's almost over.

Sorry, but memory, storage and network are all orders of magnitude cheaper today than when most of these text protocols were originally developed.

We have significantly more capacity today than we did back then. Thats why we waste all that headroom on reinventing everything in javascript.

They also tend to become interop nightmares due to having an incomplete/ambiguous/non-existent grammar. Different implementations end up implementing slightly different rules which ends up requiring an accumulation of hacks to parse all the variations that end up in the wild.

Found the healthcare IT guy.

This is the HL7 spec's issue. Everyone interprets the spec slightly differently. It's given rise to the interface engine, which are a type of very powerful software that sits between systems and makes things work properly, which is why I love them.

> They are also:

- Get funny when you need to pass binary data.

They also all eventually encode control data as text, which then causes errors with parsing some data that coincidentally has those same control characters in it.

Just look at the garble URLs you see sometimes, more percent signs than a Macy's sale.

Memory, CPU, and bandwidth are cheap. The good reason to optimize a protocol for machines is when you spend more on computers parsing plain text than on humans reading binary dumps. Most companies are not near that kind of transition.

- complicated and slow to parse

- more bloated than binary

Not neccessarily, you can write fast compact protocols with text... sending integer and float data as text is not the bottleneck in any system:

See my root comment!

Floating point numbers are INCREDIBLY complicated to scan (and print).

https://code.woboq.org/userspace/glibc/stdio-common/vfscanf-...

Compare that with reading 4 bytes directly into an ieee754 float32.

If your messages are short, the benefits of fast codecs are outweighed by the inefficiencies in the communication system (most of your time, processing power and bandwidth are taken up by setting up and tearing down the communication medium). If it takes 7 "administration" packets to send 1 data packet, your codec won't be your bottleneck (in which case you probably don't care about efficiency anyway, and this discussion is not for you).

It's not that bad, maybe 10x of nothing.

There are much bigger fish to fry when building a large network solution, most prominently getting the thing to be debuggable on live machines!

>large network solution

You basically restrict networking to big monopolists, like Google, but Google likes binary protocols, like grpc, http2 and quic. And if you have a bug in a complex parser, having text won't help debuggability, because the bug is not in text.

I make my own open-source systems, google is going down a very wrong path recently with defaulting to HTTPS and deprecating HTTP.

GRPC, HTTP2, HTTPS, WebSockets, QUIC (HTTP3) are all desperate attempts for job safety. HTTP/1.1 is good enough for 99% of human network requirements.

If you are depending on google, make sure you have alternatives because they are VERY unreliable long term.

Reply on news.ycombinator.com