Hacker News Reader

Comment by userbinator | Hacker News Reader

I suppose you could saw that parsing any text-based protocol in general "Is a Minefield". They look so simple and "readable", which is why they're appealing initially, but parsing text always involves lots of corner-cases and I've always thought it a huge waste of resources to use text-based protocols for data that's not actually meant for human consumption the vast majority of the time.

Consider something as simple as parsing an integer in a text-based format; there may be whitespace to skip, an optional sign character, and then a loop to accumulate digits and convert them (itself a subtraction, multiply, and add), and there's still the questions of all the invalid cases and what they should do. In contrast, in a binary format, all that's required is to read the data, and the most complex thing which might be required is endianness conversion. Length-prefixed binary formats are almost trivial to parse, on par with reading a field from a struture.

View on HN

Binary formats have their own serious problems.

> Length-prefixed binary formats are almost trivial to parse

They definitely are not, as displayed by the fact that binary lengths are the root cause of a huge number of security flaws. JSON mostly avoids that.

> the most complex thing which might be required is endianness conversion

That is a gross simplification. When you look at the details of binary representations, things get complex, and you end up with corner cases.

Let's look at floating point numbers: with a binary format you can transmit NaN, Infinity, -infinity, and -0. You can also create two NaN numbers that do not have the same binary representation. You have to choose single or double precision (maybe a benefit, not always). Etc.

Similarly in JSON integers or arrays of integers are nothing special. It is mostly a benefit not to have to specify UInt8Array.

JSON is one of many competitors within an ecology of programming, including binary formats, and yet JSON currently dominates large parts of that ecology. So far a binary format mutation hasn't beaten JSON, which is telling since binary had the early advantage (well: binary definitely wins in parts of the ecology, just as JSON wins in other parts).

> They definitely are not, as displayed by the fact that binary lengths are the root cause of a huge number of security flaws. JSON mostly avoids that.

I assume you're mainly referring to buffer overflows, which are a problem with text based formats too. See for example the series of overflow vulnerabilities in IIS's HTTP parser which lead to some of the most disruptive worms in history like Code Red. Really this is more of a problem with memory-unsafe languages than serialization formats.

> Let's look at floating point numbers: with a binary format you can transmit NaN, Infinity, -infinity, and -0

Depending on the use case being able to encode these values may be a requirement, in which case binary is no worse than text.

> You can also create two NaN numbers that do not have the same binary representation.

This is specific to IEEE 754, not all binary representations have this issue. Text based formats also have far more pervasive problems with lacking a canonical representation so it's hard to count this as a point against binary.

> JSON is one of many competitors within an ecology of programming, including binary formats, and yet JSON currently dominates large parts of that ecology.

This is just an appeal to popularity fallacy.

> I assume you're mainly referring to buffer overflows,

That is the most visible security issue, but there are many others e.g. reading excess data is a security flaw (a la heartbleed).

> Really this is more of a problem with memory-unsafe languages than serialization formats

JSON is often used to communicate with unsafe languages. Are you suggesting a binary format is better for using with a language that is memory-unsafe? Or are your implying we should use Rust so that we can use a binary format!

> This is specific to IEEE 754, not all binary representations have this issue.

So now we use some other (unspecified) binary format for floating point numbers?

> This is just an appeal to popularity fallacy.

Bullshit. JSON didn't become popular because it was popular. Developers have chosen to use JSON because it served a purpose for them within their particular ecology. It became popular into the headwinds of XML and other formats.

There are plenty of unpopular binary formats that developers have had experience with that they choose not to use. I have personally have enough experience with a variety of spec and custom binary formats to know when I would use one.

Binary formats most definitely have their place (embedded, high throughput at scale, severe bandwidth restrictions, strict typing, languages with poor string handling, naturally binary data). But for a large percentage of software projects, JSON works and it works well.

There is no functional difference between an array of unsigned bytes (binary) and a array of signed bytes (char data). The only difference is that when you send binary, 0 is now a valid value instead of null terminating a string. Therefore you must prepend the size because you can no longer parse till you find a NULL byte. It is always safer to know the size you must allocate ahead of time instead of dynamically growing a buffer until the text stream is terminated.

You can still reserve one value as a control character and have non-text streams that don't need prefixing of data frames by size.

These are also easier to recover the data from in case of corruption, or re-sync the receiver, since you have a control byte you can sync on.

Yes, in fact, this is most common in video streaming formats. These types of streams are more commonly downloaded as opposed to uploaded where the server has to be careful not to exhaust too many resources parsing variable-length messages.

It's also fairly common if you have alway-on streams without any signalling, where you can connect to the stream at any time, like with UART.

JSON over HTTP can communicate its size with an HTTP header.

There’s no reason strings cant be sent like binary if you do a size header first. The problem is trying to send binary like a string where your data might have 0s that could be interpreted by the receiver as string terminations. Typically base64 is used to address this issue

Even if it's compressed?

There are plenty of examples of binary formats where you do not know buffer sizes until you've received all the data, and where assumptions with parsing the data can cause a buffer overflow.

decompression and PNG libraries for example have caused massive security impact across the industry because of reuse in different products. Font handling, compressed bitmap, and windows cursor parsing also have been sources of issues.

Mozilla didn't just invest in Rust because parsing HTML and JSON are hard. Its all hard.

“It is always safer to know the size you must allocate ahead of time instead of dynamically growing a buffer until the text stream is terminated.”

And then you go on to give examples of said scenarios of how this is true while saying I’m wrong? Anytime you have an unknown payload you have to make a determination of how long you’re going to wait, how much you’re going to accept, buffer, etc before it’s become a drain on the system

I see both arguments.

XML became popular because of its human readability compared to binary and allowed disparate systems to cooperate. JSON is even more readable and allowed the whole XLST thing to be ignored, making a dev's life easier, and really took off with node.

Binary is not gone, but you don't see a lot of it in the 'web world', because everyone in that space is JSON, almost exclusively.

Having written many parsers, binary is by far and away the easiest: provided your language/environment supports it. That is, binary in js is a pain, because you don't have native ints and floats. In js, JSON is the base atomic data structure, so it makes complete sense that it's used... But in c/c++ and co. JSON is harder and introduces a lot of overhead/ quirks which simply don't exist in binary (provided you cover the buffer overruns and co.)

> XML became popular because of its human readability compared to binary and allowed disparate systems to cooperate. JSON is even more readable and allowed the whole XLST thing to be ignored, making a dev's life easier, and really took off with node.

XML was simply never designed to represent structured data. It was meant to represent document markup in a way that was both simpler and more extensible than SGML.

If there's one thing the industry always seems to do, its embrace some technology hammer as the solution for every problem. Before XML some people were trying to trade around relational database dumps as data interchange formats, because relational databases were the golden hammer.

JSON is being abused by being stretched beyond its sweet spot as well, but there aren't industry consortiums necessarily pushing bad ideas like there was with SOAP and WS-*.

JSON has JSON Object Signing and Encryption (JOSE), compare that to XML-Dsig and XML-Enc. It has OpenAPI/Swagger, which is close to SOAP/WSDL. It has JSON-Schema, which is close to XML-Schema. It has OpenID Connect, which is literally SAML, but with JSON instead of XML.

If you're missing anything out of WS-* in JSON, you can be sure somebody is working on a spec for it.

Erlang and elixir are two more memory safe languages that comfortably handle binary. See Armstrong, Pogramming Erlang, Ch. 5, "Advanced Bit Syntax Examples", with partial parsers for MPEG, COFF, and IPv4 on pages 83-89.

>This is specific to IEEE 754, not all binary representations have this issue.

Also sometimes it is a feature. The payload of a NaN value can be user defined and some programs use it[1]. The string "NaN" drops information that might be usefull to some programs, it just doesn't affect as many as null -> "null" does.

[1] https://github.com/WebKit/webkit/blob/master/Source/JavaScri...

> with a binary format you can transmit NaN, Infinity, -infinity, and -0. You can also create two NaN numbers that do not have the same binary representation. You have to choose single or double precision (maybe a benefit, not always). Etc.

Boy you are really stretching to make this sound complicated. It's not. You transmit 4 bytes or 8 bytes. Serialization is a memcpy().

You don't have to think about NaNs and Infinities because they Just Work -- unlike with textual formats where you need to have special representations for them and you have to worry about whether you are possibly losing data by dropping those NaN bits. If you want to drop the NaN bits in a binary format, it's another one-liner to do so.

It's funny that you choose to pick on floating-point numbers here, because converting floating-points to decimal text and back is insanely complicated. One of the best-known implementations of converting FP to text is dtoa(), based on the paper (yes, a whole paper) called "How to Print Floating-Point Numbers Accurately". Here's the code:

http://www.netlib.org/fp/dtoa.c

Go take a look. I'll wait.

dtoa() is not even the state of the art anymore. Just in the last few years there have been significant advances, e.g. Grisu2, Grisu3, and Dragon4...

Again, in binary formats, all that is replaced by a memcpy() of 4 or 8 bytes.

(A previous rant of mine on this subject: https://news.ycombinator.com/item?id=17277560 )

> > Length-prefixed binary formats are almost trivial to parse

> They definitely are not, as displayed by the fact that binary lengths are the root cause of a huge number of security flaws. JSON mostly avoids that.

Injection (forgetting to escape embedded text) is the root cause of a huge number of security flaws for text formats. Length-prefixed formats do not suffer from this.

What "huge number of security flaws" are you referring to that affect length-delimited values? Buffer overflows? Those aren't caused by values being length-delimited, they are caused by people deserializing variable-length values into fixed-length buffers without a check. That mistake can be made just as easily with a text format as with a binary format. In fact I've seen it much more often with text.

> JSON currently dominates large parts of that ecology.

JSON wins for one simple reason: it's easy for human developers to think about, because they can see what it looks like. This is very comforting. It's wasteful and full of pitfalls ("oops, my int64 silently lost a bunch of bits because JSON numbers are all floating point"), but comforting. Even I find it comforting.

Ironically, writing a full JSON parser from scratch is much more complicated that writing a full Protobuf parser. But developers are more comfortable with the parser being a black box than with the data format itself being a black box. ¯\_(ツ)_/¯

(Disclosure: I am the author of Protobuf v2 and Cap'n Proto. In addition to binary native formats, both have text-based alternate formats for which I wrote parsers and serializers, and I've also written a few JSON parsers in my time...)

Those aren't caused by values being length-delimited, they are caused by people deserializing variable-length values into fixed-length buffers without a check. That mistake can be made just as easily with a text format as with a binary format. In fact I've seen it much more often with text.

Especially since text can be arbitrarily long. From that perspective, length-delimited text (I've seen that before in a few file formats, and more notably HTTP) is probably the worst of both worlds.

I think people undervalue clean-looking (alphabet-only, few special character) things, things that don't require people to use the symbol-parsing part of their brain. Basically easily human-parseable things. I suspect this phenomenon can be observed in the case of relative popularity of JSON, TOML, YAML and Python plus the relative unpopularity of Lisp, Haskell, Rust and XML. And if we look at protobuf in this context it is not easy to parse for humans, which causes people not to want to use it, developers are not

> more comfortable with the parser being a black box

they're more comfortable with the parser being a black box but the format being relatively easy to parse compared to the parser being easy to understand but the format basically unreadable for a human.

> I think people undervalue clean-looking (alphabet-only, few special character) things, things that don't require people to use the symbol-parsing part of their brain. Basically easily human-parseable things.

The symbol parsing part of the human brain is what parses letters and numbers, as well as other abstract symbols. The division of symbols into letters, numbers, and others is fairly arbitrary. Most people would say “&”, but the modern name of that symbol is a smoothing over of the way it was recited when it was considered part of the alphabet and recited with it.

> I suspect this phenomenon can be observed in the case of relative popularity of JSON, TOML, YAML and Python plus the relative unpopularity of Lisp, Haskell, Rust, XML.

I suspect not: Lisp and Haskell have less use of non-alphanumeric characters than most more-popular general purpose languages, and not significantly more than Python; also, if this was the phenomenon in play, popularity would be TOML > YAML > JSON but in reality it's closer to the reverse.

> The symbol parsing part of the human brain is what parses letters and numbers, as well as other abstract symbols.

I really don't think that's true when you talk about about someone using the latin alphabet, words in that alphabet compared to some other alphabet (e.g. {}():!) and "words" (or meanings) in those. Just as a crude example parsing "c = a - b", where equals and minus are one symbol each and have been taught for a while, is different from parsing "c := a << b" where ":=" and "<<" basically act as a separate meaning someone has to learn to understand. Similar to the difference of latin alphabet and say simplified Chinese.

> also, if this was the phenomenon in play, popularity would be TOML > YAML > JSON but in reality it's closer to the reverse.

There could be somewhat of an sigmoid response to the effect, decreased reaction if you go into either extreme compared to deviating from the average.

I'm not a linguist so it is my speculation, so don't take it too seriously :D

That's exactly what I was trying to say. Sorry if I wasn't clear.

> Similarly in JSON integers or arrays of integers are nothing special.

JSON is perfectly capable of representing integers which cannot be represented in IEEE-754 double precision floating point. That seems at least a little special to me.

> You can also create two NaN numbers that do not have the same binary representation.

Way worse than that. The interpretation of those different values of NaN is software specific. You can also have signalling NaN values - where the recipient can now have their number handling code trap in completely unexpected scenarios.

Consider only this: "1.001"

I'll use JavaScript numeric literals here as my translation medium (ironic!):

Norway locale parses it to: 1001

USA locale parses it to: 1.001

France locale parses it to: NaN

https://docs.oracle.com/cd/E19455-01/806-0169/overview-9/ind...

No. In a programming context, any norwegian or french programmer would expect that to evaluate to 1.001, not 1001 or NaN.

Sweet summer child, isn't that what people say these days?

I and you can agree that the reasonable thing to do is to accept American encoding.

(The below is somewhat simplified, the way I remember it on a late Saturday night 10+ years later.)

The outsourced team of programmers from our software vendor did not. They (mostly) used the built in regional settings in Windows (best practice, don't reinvent the wheel) meaning we had to come up with ways to make sure the machines ran with wrong regional settings (since a lot of stuff was already serialized that way && critical parts of the software was hardcoded to use US standard.)

Fun times ;-)

I remember Excel also has(had?) some very disturbing behaviour around the interaction of CSV files and the regional decimal point separator settings.

Not sure about CSV files, but every time I'm pasting numbers generated by Python or Java I had to use Notepad or Excel itself to replace dots to commas because of (Windows'!) regional settings.

In a somebody sent this input to my program context, designing some means to choose between these three results (or possibly others) is the main concern. Actual parsing is the easy part.

Do you get the user to choose? Great, now the UI is 2x more complex.

Do you just go with what the browser or client locale says? (User lives in Canada but their laptop is set to Turkish locale).

Go with the locale tied to the geo location of the user instead? (User above stubbornly enters all amounts using Norwegian conventions after completing high school in Norway).

If it's important, a confirmation page presented to the user and formatted in their presumed locale can help a lot.

This is a little too generous about the benefits of binary formats vs text formats. Ultimately, any data exchange between disparate systems is going to be a challenging task, no matter what format you choose. Both sides have to implement it in a compatible way. And ultimately, every format is a binary format. Encoding machine-level data structures direct on the wire sounds good, but it quickly gets complicated when you have to deal with multiple architectures and languages. And you don't have the benefit of the gradually accreted de-facto conventions like using UTF-8 encoding for text-based formats to fall back on, much less the ability for humans to troubleshoot by being able to read the wire protocol.

With sufficient discipline and rigor, and a good suite of tests, developed over years of practical experience, you can evolve a good general binary wire protocol, but by then it will turn out to be so complicated and heavyweight to use, that some upstart will come up with a NEW FANTASTIC format that doesn't have any of the particular annoyances of your rigorous protocol, and developers will flock to this innovative and efficient new format because it will help them get stuff done much faster, and most of them won't run into the edge cases the new format doesn't cover for years, and then some of them will write articles like this one and comments like yours and we can repeat the cycle every 10-20 years, just like we've been doing.

Wait wait. XML with XSD Schemas are a easy problem. You can't fail with a XSD schema on place

>Consider something as simple as parsing an integer in a text-based format; there may be whitespace to skip, an optional sign character, and then a loop to accumulate digits and convert them (itself a subtraction, multiply, and add), and there's still the questions of all the invalid cases and what they should do.

    ^[ *][-?][0-9][0-9]*[ *]$

You're welcome. Anything that passes that regex is a valid number. Now using that as a basis of a lexer means that you can store any int in whatever precision you feel like.

It's unfortunate that the majority of programmers these days are so computer illiterate that they can't write a parser for matching parens and call you an elitist for pointing out this is something anyone with a year of programming should be able to do in their sleep.

Just matching that regex alone is going to take a pretty large number of instructions (at least dozens.) That's not "simple" by any measure.

Probably tens of thousands.

And it will still be a faster operation than fetching the next json file from disk by an order of magnitude, or two.

Reply on news.ycombinator.com