Comment by verisimilitudes | Hacker News Reader

Comment by verisimilitudes | Hacker News Reader

Parent Full thread

verisimilitudes

So, how long until everything is JSON-shit over port 80 and how long until port 80 is old news and only port 443 is used for anything and everything? I can't claim to be familiar with IMAP, but it can't be worse than this garbage that requires an HTTP client and a JSON parser.

To process JSON, you must inspect every character, which is reason enough it's an awful ''format'' that only an ignorant could hold dear. Douglas Crockford brags about how smart he is for disallowing comments, but there's whitespace allowed everywhere. Let's not forget this asinine integer nonsense, because JavaScript lacks real integers.

I'm sure most people think HTTP clients and JSON parsers are standard-library affairs, nowadays, though; let's just entirely ignore matters of licensing and whatnot.

It's disgusting to see this as an IETF standard, but people have been complaining about how that moves along for decades; take a look at Google's robots.txt ''standard'' that doesn't do anything useful and effectively makes every existing file standard, rather than do much of anything worthwhile. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth that UNIX and C previously championed; decades ago, the IETF revised the RFC for the Finger protocol and warned about security, solely because some idiot wrote a shitty Finger server in C; now, we have useless and unnecessary standards such as this that only serve to further this next generation of idiocy.

I wonder when all of this will finally collapse under its own weight.

Can you provide some actual arguments as to what's wrong with JSON? It's a text format that can be inspected by humans, unlike binary formats. (Maybe IMAP is a text format too, so this isn't an advantage inherently) And JMAP may have better semantics than IMAP, and there's nothing inherently wrong with JSON (other than edge cases).

The problems with JSON are pretty obvious:

* It's inefficient compared to schema-based formats like Protobuf - you have to encode key names again and again.

* It's inefficient compared to binary equivalent formats like CBOR - both in space and speed.

* It's less reliable than explicit schema-based formats because it can't be validated (yes I know about JSON-Schema, nobody uses it). You essentially end up with an implicit schema that everyone has to recreate themselves from the documentation - especially if using statically typed languages.

* It's a pain to use from statically typed languages. You have to write the schema in your language, whereas with things like Protobuf it will generate the code for you.

* It doesn't have a proper type for arbitrary binary data. Most people (but not all!) encode it as a base64 string which is not very efficient and not actually a different type to a string, so again you rely on the implicit schema.

* Maps can only have strings as their key type.

* Minor annoyance, but trailing commas would be nice!

* Its number type is not well specified. E.g. does it support uint64?

"Human readable" is overrated. CBOR and similar formats can easily be inspected by humans just by converting them to JSON. The Smile format even starts with a :) smilie as its magic number to make it easily recognisable.

Would you recommend not using compression or encryption on HTTP requests because it stops them being human readable?

I use json-schema. I hate it but, i use it.

>Can you provide some actual arguments as to what's wrong with JSON?

It's a wasteful, textual format derived from a programming language vomited forth in under a week. It has asinine integer characteristics for no good reason; it's not a random-access format; validating it is even more work; hash table keys that appear multiple times, instead of being in error, instead overwrite each other, defeating most any kind of optimization; it is thus wildly inefficient to manipulate, as manipulating it correctly in all cases requires full and comprehensive parsing with checking. It is an asinine ''format''.

>It's a text format that can be inspected by humans, unlike binary formats.

Even then, there are much better textual formats that could be used, but I'm of the opinion that numerical (binary) formats are strictly superior, especially as they're used increasingly often. It's better to have good tools that will provide a human-readable representation, rather than pretend a single, inefficient representation for both man and machine is somehow the best option.

>(Maybe IMAP is a text format too, so this isn't an advantage inherently) And JMAP may have better semantics than IMAP, and there's nothing inherently wrong with JSON (other than edge cases).

Computer programs written by anyone competent are built around edge cases; that's much of what any reliable program is, accounting for edge cases. To disregarded edge cases, of which there are many in JSON, speaks to me that you have little concern for writing software correctly, but I imagine you'd deny this, as who doesn't want to write correct software? Simply understand that using a format with so many edge cases and other defects as JSON is foolhardy for much of anything.

>numerical (binary) formats are strictly superior, especially as they're used increasingly often. It's better to have good tools that will provide a human-readable representation, rather than pretend a single, inefficient representation for both man and machine is somehow the best option.

I'm working on a chiptune music composing program (FamiTracker), which saves projects in a binary format designed by my predecessor. I once repaired a corrupted file for someone else. It's borderline impossible to repair binary formats by hand, since inspecting the data on-disk provides no clues about the structure, which I don't know since I didn't write the program. I ended up merging the corrupted file and an older backup in Audacity (an audio editor) by sliding the two files back and forth relative to each other, because hex editors are awful at visualizing and aligning binary files (whereas diff tools are good at visualizing and aligning text files).

Additionally, the on-disk layout changes between versions, and the reader code is a mess of branching between different on-disk layouts based on the version number.

Additionally, it's impossible to check in your music project (binary blob) into Git, then get a meaningful diff between different versions, let alone being able to merge changes made in 2 branches. (Yes, I use Git for music. It's actually useful.)

Needless to say, I'm using a text format if I ever write a replacement. A text format may have trouble encoding binary blobs, but I don't like the idea of "transcoding .wav into binary files, and throwing away the parameters used". I'm more inclined to just use pointers to on-disk .wav, and store all metadata needed to reproduce the binary blobs, and possibly cache the binary blobs in base64 for speed and convenience.

I'm working on a chiptune music composing program (FamiTracker), which saves projects in a binary format designed by my predecessor. I once repaired a corrupted file for someone else. It's borderline impossible to repair binary formats by hand, since inspecting the data on-disk provides no clues about the structure, which I don't know since I didn't write the program.

That's interesting. If you're interested in discussing that more at some point, email me. Now, I've designed a numerical format for metadata involving machine code programs and, while I've not written this piece yet, an important aspect of this will be a repairing tool. The only reason a textual format would be easier to repair by hand is because you can use a text editor and it's more redundant. Now, good design is another quality; ideally, a numerical format would lack holes that are invalid states and wouldn't be riddled with pointers or other things if feasible; it varies based on what you're doing, though.

I ended up merging the corrupted file and an older backup in Audacity (an audio editor) by sliding the two files back and forth relative to each other, because hex editors are awful at visualizing and aligning binary files (whereas diff tools are good at visualizing and aligning text files).

That's again a simple matter of tooling that could be corrected.

Additionally, the on-disk layout changes between versions, and the reader code is a mess of branching between different on-disk layouts based on the version number.

A runaway format revision like that isn't ideal, I can agree. You can get the same mess with textual formats, however.

Additionally, it's impossible to check in your music project (binary blob) into Git, then get a meaningful diff between different versions, let alone being able to merge changes made in 2 branches. (Yes, I use Git for music. It's actually useful.)

I don't use git at all, so this is another difference between us. I'm largely of the opinion that the focus on textual formats for UNIX-likes was due to sloth more than anything, as there's generally a lack of coherency between /etc/fstab to /etc/passwd and so on; I believe the focus was largely due to the advantage of ostensibly generic tooling, but reading or writing a configuration in a text editor, with the manual handy, and getting errors and other things upon loading it is worse than having a tool write it for you and show it to you at a high-level.

Needless to say, I'm using a text format if I ever write a replacement. A text format may have trouble encoding binary blobs, but I don't like the idea of "transcoding .wav into binary files, and throwing away the parameters used". I'm more inclined to just use pointers to on-disk .wav, and store all metadata needed to reproduce the binary blobs, and possibly cache the binary blobs in base64 for speed and convenience.

I don't know the details of what you're dealing with, but again, if you're interested, email me and I can perhaps work on a repair tool or help design a replacement numerical format that will be easier to manipulate.

The problem with being clever and efficient is that you wind up having to write things like:

https://tools.ietf.org/html/rfc6868

Which is the kind of horror case of trying to do everything with clever little pieces of micro-format.

Now, the argument between XML and JSON is less of a big deal, except XML gets wishy-washy with whitespace in some ways which frustrate me. JSON number handling is screwy, it's true. I'd be tempted by CBOR if there were no other considerations - and I'd love to write a JMAP-over-CBOR or JMAP-over-ASN.1 or whatever at some point.

But you can very easily structure non-binary data in and out of JSON reliably, and everyone can get a parser which works. That's a big win. It's fine to disregard edge cases if you stay away from the edges - for example if you're serialising integers between 0 and 1000 then the fact that your datatype isn't 256 bit clean isn't a big deal. Same way you can use Newtonian physics when dealing with sprinters - but I don't think that race organisers have little concern for running a race correctly because they don't factor in relativity.

So we kept JMAP in the safe subset of JSON, because it's the best format for a whole lot of other reasons.

Specifically, JMAP explicitly requires I-JSON (https://tools.ietf.org/html/rfc8620#section-1.5).

If, for example, you try sending a request with duplicated keys in an object, the JMAP server will reject your request with the urn:ietf:params:jmap:error:notJSON error.

> hash table keys that appear multiple times, instead of being in error, instead overwrite each other

I don't think that's true. The behavior with multiple keys is not defined. A JSON reader can choose to do whatever it wants (error, use first, use last, use all, etc). An interoperable JSON writer should never output duplicate keys.

You won't believe what happened next: RFC 6902 comes along and explicitly prohibits multiple keys. Programmers want to reuse JSON parsers, but can't. This is not a good position to be in.

RFC 6902 is not a standard for JSON, it's a standard for "JSON patch", which appears to be another data format layered on top of JSON. It has no descriptive power on JSON in general.

The relevant RFC is 8259, which neither explicitly allows nor prohibits duplicate keys. Instead it says:

> An object whose names are all unique is interoperable in the sense that all software implementations receiving that object will agree on the name-value mappings. When the names within an object are not unique, the behavior of software that receives such an object is unpredictable. Many implementations report the last name/value pair only. Other implementations report an error or fail to parse the object, and some implementations report all of the name/value pairs, including duplicates.

That isn't a prohibition on duplicate keys, it's just a warning on what will happen in practice if you try to do that.

JMAP explicitly requires I-JSON, which is RFC 7493, and restricts this exact case:

> Objects in I-JSON messages MUST NOT have members with duplicate names.

Thanks! I was commenting just from the perspective of JSON in general, didn't check the JMAP spec.

(Does JMAP require validating that the input is valid I-JSON? Or does it just say that a JMAP endpoint is allowed to assume it's I-JSON, but doesn't need to validate it?)

If the input is not valid I-JSON, a JMAP server that does not reject it is non-compliant.

(Related: Postel’s law is wrong. See https://tools.ietf.org/html/draft-iab-protocol-maintenance-0... for some explanation.)

> (Maybe IMAP is a text format too, so this isn't an advantage inherently)

Actually, IMAP is a binary format that looks like a text format. This distinction is extremely important when you want to make a reliable Email client. There are also multiple ways of encoding text within it, that you need to keep track of.

Personally, I wish they used a structured binary encoding. This becomes especially important when data size efficiency actually matters. With something like protobuf, you can even have the parser for it generated for you. (However, I have worked with binary formats that are more loosely structured.)

Of course I am a fan of well defined and strongly typed formats.

- Anything outside of int32 must be encoded and decoded as a float. Oh the bugs/crashes this has caused.

- It is way too easy to change the schema and wreak havoc with downstream users.

- Some libraries will encode empty strings as null for instance, some as empty strings. Some will have empty objects, some null. Switch some configuration, change your serializer, ORM or anything that touches your data. Everything will work perfectly in your code, but json output can change and you will again break your downstream users.

If you are designing an API, use protocol buffers. Json maybe as an optional output/input to your service. Keep it as a discouraged dirty optional feature and you will save yourself a whole lot of debugging and time wasting.

google defined websocket protocol not long ago. We didn't actually need it. They did. With such a huge browser market share, all they crave for is information. All of it. And most of us (developers) embrace this flow because each and every "fault" they "discover" and use to create a new idiotic standard is an opportunity for us to profit. And we feed it. I don't need jmap, none of us do. They need it, so they wrote 200 pages of stupid, extremely complicated specs with the hope that it will gain traction and we'll start developing. They can't do it alone. They need us to embrace it. I won't and I hope none of you will. I hope this standard will die with a whimper. I am doing my best to keep my e-mails where they belong. Far away from browser. I use SMTP. There's nothing wrong with it. It works. I use MIME. There's nothing wrong with it. It works. I personally don't use IMAP. ODMR is all I actually need. But I configure and maintain IMAP servers. I see nothing wrong with it either. It works. What baffles me is cyrus project accepting this idiocy. I just hope it won't become the new deal and the others (dovecot et all) will stay away from it. Panic and fear can and will escalate the process.

So how long until "everything is JSON"-shit will take over? I'm 40 years old and my bet is I will live to see it happening. This is actually scary. And not because json or hydras.

To be honest, JSON is pretty reasonable. The parsing speeds are very good, the grammar is a lot simpler than SMTP/HTTP/IMAP, it actually has numeric and boolean types.

It is no surprise to me that the two people here complaining that IMAP is somehow acceptable don't even use it.

JMAP solves real problems inherent with IMAP, particularly on clients that are not suitable for local archiving of messages (hence, ODMR/POP is not an option), and where IMAP sends considerably more data than the client actually uses (making it slow).

I don't like the use of JSON for local IPC, as is showing up in some systems, but for RPC over an unreliable network, JSON is more convenient than not.

As for the use of HTTP, it's not ideal in my view, and I tend to think it's overused. The one argument though, is that it allows JMAP to go over port 443 to a host that has an existing https service on that port (since you likely have a reverse proxy, probably NGINX, as your listener anyway) without having to send a bespoke upgrade request, which defeats the simplicity benefits of a custom protocol, since you have to implement HTTP to do the upgrade.

I tend to agree with you, even so, if not for these moves we wouldn't have any change at all. Email is not a solved problem, we just have a lot of fixes that makes it sort of stable. If not else we get a competing extension to IMAP because someone does not want JMAP.

Reply on news.ycombinator.com