Hacker News Reader

Comment by nayuki | Hacker News Reader

Oddly enough, BitTorrent's bencode format is a significant subset of JSON but is much easier to parse as a binary format. Bencode only supports integers, byte strings, lists, and dictionaries.

I wrote a more detailed comparison on: https://www.nayuki.io/page/bittorrent-bencode-format-tools

View on HN

"BitTorrent became popular around year 2005, whereas JSON became popular around year 2010. The usage of JSON in the real world has greatly eclipsed BitTorrent or bencode, so there is a natural bias to view bencode through the lens of JSON even though JSON was adopted later (though not necessarily invented later)."

Netstrings was proposed in 1997.

https://cr.yp.to/proto/netstrings.txt

Whether bencoding's remarkable similarity to netstrings is purely coincidence is left as a question for the reader.

Perhaps there is a "natural bias" to view netstrings through the lens of bencoding even though bencoding came much later.

"Oddly enough, BitTorrent's bencode format is a significant subset of JSON but is much easier to parse as a binary format."

Curious what makes it "odd".

JSON assumed that memory is an unlimited resource.^1 It is like having to read an entire file into memory before processing it. Hence we see revisions such as "line-delimited JSON". Netstrings is even more memory-efficient than line-oriented processing; there is no need to read in an entire line.

1. This makes sense if one agrees that JSON was designed for graphical web browsers, programs notorious for excessive memory usage.

tnetstrings[1] was a later refinement of netstrings.

I liked it, but alas it never seemed to really take off.

[1]: https://tnetstrings.info/

So if you want strings, you need to guess what encoding was used or store the encoding in another field? I don't think that makes it a much nicer format. I do like the ability to store byte strings directly.

Oh, I didn't know about Bencode. It looks interesting. Thank you for sharing!

I really like bencode. The only thing I miss is floats.

You can use two integers, one that represents the entire number including decimals, and one that represents the precision, to know how many decimals are there. For example, you'd represent "123.45" as 12345 and 2. That's often how monetary amounts are stored in databases, to avoid a lot of common floating-point arithmetic pitfalls.

But that's just floats with extra steps? Floats have two parts in their binary representation: mantissa and exponent (and sign), which correspond exactly to your "entire number" and "precision", only in base 2 instead of 10.

the difference being that with integers you never end up with rounding errors when doing addition subtraction or multiplication (only division)

Or just '123' & '45'?

I think it is to optimize arithmetic operations. Significantly less steps with the first method, which only requires adjustment of how many digits are considered to be decimal rather than the rejoin, arithmetic, separate again for your proposal. Plus, wider float.

But then you can't tell the difference between 0.12 and 0.00012.

Unless you're suggesting to use the strings "0" and "00012", at which point you could just use a byte string with the utf8 encoding of the value.

Reply on news.ycombinator.com