Why I stopped using JSON for my APIs
aloisdeniel.com
aloisdeniel.com
This deeply misunderstands the philosophy of Protobuf. proto3 doesn't even support required fields. https://protobuf.dev/best-practices/dos-donts/
> Never add a required field, instead add `// required` to document the API contract. Required fields are considered harmful by so many they were removed from proto3 completely.
Protobuf clients need to be written defensively, just like JSON API clients.
Most of my APIs are internal APIs that accept breaking changes easily. My experience with protobufs is that it was created to solve problems in large systems with many teams and APIs, where backwards compatibility is important. There are certainly systems where you can't "just" push through a breaking API change, and in those cases protobufs make sense.
Also significant distribution such that it’s impossible to ensure every system is updated in lockstep (at least not without significant downtime), and high tail latencies e.g. a message could be stashed into a queue or database and processed hours or days later.
I went all in with Go's Marshalling concept, and am using my Gooey framework on the client side nowadays. If you can come around Go's language limitations, it's pretty nice to use and _very_ typesafe. Just make sure to json:"-" the private fields so they can't be injected.
[1] shameless drop https://github.com/cookiengineer/gooey
You could also just validate the JSON payload, but most people don't bother. And then they just pass the JSON blob around to all sorts of functions, adding, modifying, and removing fields until nobody knows for sure what's in it anymore.
I don't think I have ever worked somewhere that didn't require people to validate inputs.
The only scenario could be prototypes that made it to production, and even when its thrown over the wall I'll make it clear that it is unsupported until it meets minimum requirements. Who does it is less important than it happening.
Usually it's not super strict, as in it won't fail if a new field suddenly appears (but will if one that's specified disappears), but that's a configuration thing we explicitly decided to set this way.
I didn't find details of the caching design. Possibly it was mentioned to me by waddlesplash on IRC[3].
[1] 500 Byte Images: The Haiku Vector Icon Format (2016) http://blog.leahhanson.us/post/recursecenter2016/haiku_icons...
[2] Why Haiku Vector Icons are So Small | Haiku Project (2006) https://www.haiku-os.org/articles/2006-11-13_why_haiku_vecto...
[3] irc://irc.oftc.net/haiku
Indeed, would be nice if one of these blogs explained the caching solution to tackle the drawback.
Another issue, I think, especially at smaller sizes, is the pixel snapping might be imperfect and require "hints" like in fonts? Wonder if these icons suffer from these/address it
That's the point: obliviousness to different concerns and their importance.
Among mature people, the main reason to use SVG is scaling vector graphics (in different contexts, including resolution-elastic final rendering, automatically exporting bitmap images from easy to maintain vector sources, altering the images programmatically like in many icon collections); worrying about file sizes and rendering speed is a luxury for situations that allow switching to bitmap images without serious cost or friction.
Oof. I'd rather just version the endpoints and have required fields. Defensive is error-prone, and verbose, harder to reason about, and still not guaranteed. It really feels like an anti-pattern.
Sure it will blow up in your face when a field goes missing or value changes type.
People who advocate paying the higher cost ahead of time to perfectly type the entire data structure AND propose a process to do perform version updates to sync client/server are going to lose most of the time.
The zero cost of starting with JSON is too compelling even if it has a higher total cost due to production bugs later on.
When judging which alternative will succeed, lower perceived human cost beats lower machine cost every time.
This is why JSON is never going away, until it gets replaced with something with even lower human communication cost.
that's true. But people also rather argue about security vulnerabilities than getting it right from the get-go. Why spend an extra 15 mins effort during design when you can spend 3 months revisiting the ensuing problem later.
I once converted a fairly large JS codebase to TS and I found about 200 mismatching names/properties all over the place. Tons of properties we had nulls suddenly started getting values.
Although, an esoteric language defined in terms of negative space might be interesting. A completely empty source file implements “hello world” because you didn’t write a main function. All integers are incremented for every statement that doesn’t include them. Your only variables are the ones you don’t declare. That kind of thing.
Oh I forgot to qualify that I'm only talking about high level code, not things that you'd use C or Rust for. But part of the reason those langs have static types is they need to know sizes on stack at compile time.
Yup this is it. No architect considers using protos unless there is an explicit need for it. And the explicit need is most times using gRPC.
Unless the alternative allows for zero cost startup and debugging by just doing `console.log()`, they won't replace JSON any time soon.
Edit: Just for context, I'm not the author. I found the article interesting and wanted to share.
And since you need to translate it anyway, there's not much benefit in my mind to using something like msgpack which is more compact and self describing, you just need a decoder to convert to json when you display it.
I'm not guessing. I'm using my knowledge of the program and the error together to decide what to print. I never find the process laborious and I almost always get the right set of variables in the first debug run.
The only time I use a debugger is when working on someone else's code.
And that lines up with some of the appeals to authority there that are good, and that are bad (edited to be less toxic)
The logger vs debugger debate is decades old, with no argument suggesting that the latter is a clear winner, on the contrary. An earlier comment explained the log debugging process: carefully thinking about the code and well chosen spots to log the data structure under analysis. The link I posted was to confirms it as a valid methodology. Overall code analysis is the general debugging skill you want to sharpen. If you have it and decide to work with a debugger, it will look like log debugging, which is why many skilled programmers may choose to revert to just logging after a while. Usage of a debugger then tends to be focused on situations when the code itself is escaping you (e.g. bad code, intricate code, foreign code, etc).
If you're working on your own software and feel that you often need a debugger, maybe your analytical skills are atrophying and you should work on thinking more carefully about the code.
So the skill of knowing how to "println" debug is still very useful.
You (a human) can just open a JSON request or response and read what's in it.
With protobuf you need to build or use tooling that can see what's going on.
One can imagine a world where the lowest common denominator format being a richer structured binary format where every system has tooling to work with it out of the box and that would be considered human readable.
Unless your servers and clients push at different time, thus are compiled with different versions of your specs, then many safety bets are off.
There are ways to be mostly safe (never reuse IDs, use unknown-field-friendly copying methods, etc.), but distributed systems are distributed systems, and protobuf isn't a silver bullet that can solve all problems on author's list.
On the upside, it seems like protobuf3 fixed a lot of stuff I used to hate about protobuf2. Issues like:
> if the field is not a message, it has two states:
> - ...
> - the field is set to the default (zero) value. It will not be serialized to the wire. In fact, you cannot determine whether the default (zero) value was set or parsed from the wire or not provided at all
are now gone if you stick to using protobuf3 + `message` keyword. That's really cool.
this seems like a problem only if you use JSON.parse or json.loads etc. and then just cross your fingers and hope that the types are correct, basically doing the silent equivalent of casting an "any" type to some structure that you assume is correct, rather than strictly parsing (parse, don't validate) into a typed structure before handing that off to other code.
That's called validating? Zod is a validation library.
But yeah, people really need to start strictly parsing/validating their data. One time I had an interview and I was told yOu DoN'T tRuSt YoUr BaCkeNd?!?!?!?
looking at zod (assuming https://zod.dev) it is a parsing library by that definition — which isn't, like, an official definition or anything, one person on the internet came up with it, but I think it is good at getting the principle across
under these definitions a "parser" takes some input and returns either some valid output (generally a more specific type, like String -> URL) or an error, whereas a "validator" just takes that input and returns a boolean or throws an error or whatever makes sense in the language.
eta: probably part of the distinction here is that since zod is a JS library the actual implementation can be a "validator" and then the original parsed JSON input can just be returned with a different type. "parse don't validate" is (IMO) more popular in languages like Rust where you would already need to parse the JSON to a language-native structure from the original bytes, or to some "JSON" type like https://docs.rs/serde_json/latest/serde_json/enum.Value.html that are generally awkward for application code (nudging you onto the happy parsing path).
I have not found it difficult to support backwards compatibility with explicit versioning, and the only justification I’ve seen for not doing it is that it’s impossible to coordinate between development teams. Which I think is an indictment of how the company is run more than anything else.
Searched the article, no mention of gzip, and how most of the time all that text data (html, js and css too!) you're sending over the wire will be automatically compressed to...... an efficient binary format!
So really, the author should compare protobufs to gzipped JSON
But in the end compressed JSON is pretty good. Not perfect, but good enough for many many things.
NOT worth it, especially if the whole infra is already using JSON.
> This can sound like nothing, but considering that Protobuf has to be converted from binary to JSON - JavaScript code uses JSON as its object literal format - it is amazing that Protobuf managed to be faster than its counterpart.
Presumably the difference would be much larger for languages that can actually represent a statically-typed structure efficiently.
Also, the tradeoffs have changed since Protobuf was invented. Network bandwidth has gotten cheaper faster than CPU bandwidth has, so the en/de-coding speed is more important than the packet size in many situations. And if you don't use gzip, Protobuf is much faster (especially in non-JS languages, and especially if you use fixed-size integer types instead of variants).
If your data consists of 4 kilobytes of just 00_01, then you gain a lot by just remembering:
"write 00_01 2000 times".
"write 00_01 3 times"
Protobuf has advantages, but is missing support for a tons of use cases where JSON thrives due to the strict schema requirement.
A much stronger argument could be made for CBOR as a replacement for JSON for most use cases. CBOR has the same schema flexibility as JSON but has a more concise encoding.
But yes, a native implementation would save me the trouble!
[ciborium]: a Rust CBOR library; https://docs.rs/ciborium/latest/ciborium/
https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER
Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves.
(I wrote a implementation of DER encoding/decoding in C, which is public domain and FOSS.)
After working heavily with SNMP across a wide variety of OEMs, this flexibility becomes a downside. Or SNMP/MIBs were specified at the wrong abstraction level, where the ASN.1 flexibility gives mfgs too much power to do insane and unconventional things.
Of course, I am an ASN.1 compiler maintainer, but hey, I had to become one because the compiler I was using was awesome but not good enough, so I made it good enough.
I'm curious what made it a nightmare for you.
I also remember it being complicated to use, but it's been too long to recall why exactly, probably the feature bloat. Once I used proto3, I realized it's all you need.
Only if you want to implement them. You could get quite far with just a subset of UNIVERSAL types, including UTF8String, SEQUENCE/SET, SEQUENCE OF / SET OF, etc. There's a ton of features in x.680 you can easily drop.
I've implemented a subset of x.681, x.682, and x.683 to get automatic, recursive decoding through all typed holes in PKIX certificates, CRLs, CSRs, etc. Only a subset, and it got me quite far. I had a pretty good open source x.680 implementation to build on.
This is the story of how Heimdal's authors wrote its ASN.1 compiler: they wanted tooling, there wasn't a good option, they built enough for PKIX and Kerberos. They added things as they went along. OpenSSL does not-quite-DER things? Add support in the Heimdal decoder. They hacked a lot of things for a while which I later fixed, like they didn't support DEFAULT, so they changed DEFAULTed members to OPTIONAL, and they hacked IMPLICIT support, which I finished. And so on. It still doesn't have things like REAL (who needs it in security protocols? no one). Its support for GeneralString is totally half-assed just like... MIT Kerberos, OpenSSL, etc. We do what we need to. Someone could take that code, polish it up, add features, support more programming languages, and make some good money. In fact, Fabrice Belllard has his own not-open-source, commercial ASN.1 compiler and stack, and it must be quite good -- very smart!
- PB being a TLV encoding, just like DER, with all the same problems
(Instead PB should have been inspired by XDR or OER, but not DER.)
- PB's IDL requiring explicitly tagging every field of every data structure(!) even though ASN.1 never required tagging every field, and even though ASN.1 eventually adopted automatic tagging.
- PB's very naive approach to extensibility that is just like 1984 ASN.1's.
Some people, when faced with a dearth of tooling, will write said tooling. Other people will say that the technology in question is a nightmare, and some of those people will then go on to invent a worse wheel.
I'd be ecstatic to use something other than ASN.1 if it wasn't a poor reinvention of it.
Many modern high-volume telemetry systems use gRPC for a good reason, it wins in the "pragmatic" department.
Also I don't see special ASN1 support for non-Unicode string encodings, only subsets of Unicode like ascii or printable ascii. It's a big can of worms once you bring in things like Latin-1.
Also, DER allows to indicate the type of data within the file (unless you are using implicit types). Protobuf has only a limited case of this (you cannot always identify the types), and it requires different framing for different types. However, DER uses the same framing for all types, and strings are not inherently limited to 2GB by the file format.
Furthermore, there are other non-scalar types as well.
In any of these cases, you do not have to use all of the types (nor do you need to implement all of the types); you only need to use the types that are applicable for your use.
I will continue to use ASN.1; Protobuf is not good enough in my opinion.
I want to object that anyone who really has to handle multi-codeset GeneralString values would want a better library but...
...what would that be if not a converter to/from Unicode? But what if one really wants an array/list of {codeset, string} pairs? At that point open-coding support for those escapes is probably just as well since that one might have the only application in the world that wants that!!
:laugh:
You can do that for all types, and my own implementation does just expose the payloads for all types, although it also has functions to encode and decode many of them, they are functions that must be called separately (e.g. asn1_decode_number and asn1_decode_date).
For some applications, there is no need to decode the payload anyways, and you can just treat it as opaque data if you do not need to display them (the same is true for many other types).
> I want to object that anyone who really has to handle multi-codeset GeneralString values would want a better library
I did start to try to write such a thing (it is not published yet), and the library for ISO 2022 is separate than the library for ASN.1, although they can be used together. It is intended to be usable for multiple uses. (I might also add support for character codes other than ISO 2022 (such as the encodings of Unicode and TRON code), although it is mainly intended to support ISO 2022.)
> ...what would that be if not a converter to/from Unicode?
For one thing, not all of the codes can be correctly converted to/from Unicode (especially control characters), and even if they can be, this is does not preserve some of the details, because the way the character set works is different from Unicode (e.g. some things might be considered to match in Unicode but not other character sets and vice-versa; this would also be true of case-folding, missing details, character properties, etc). For example, some information may be lost when converting to Unicode.
(This does not necessarily mean that converting to/from Unicode is never useful, although you should consider if you can do it in a better way; for example, if your program accepts input that is meant to be added to a General string (or Graphic string) in a DER file then it would be better to accept ISO 2022 input directly if possible. Converting to TRON code (especially for CJK text) might be better, but even that depends on what you are using it for; some uses do not require conversion at all.)
If you really want to store Unicode text anyways, you should consider if UTF-8 is a valid type according to the schema you are using (and use that type instead if so; note that some schemas might not care so much about the type in some cases); if not, consider prepending the three bytes <1B 25 47> when encoding it as a ISO 2022 string. (As far as I can tell, this is not really supposed to be allowed in ASN.1, but I suppose it is possible if you really need to. You might also check if it is only ASCII and avoid adding this prefix if so.)
> But what if one really wants an array/list of {codeset, string} pairs? At that point open-coding support for those escapes is probably just as well since that one might have the only application in the world that wants that!!
I think that what you will want will depend on the specific application. Some will want that, some will want something else, and also different ways that you might want handling control characters, etc.
There is C0, G0, C1, and G1 sets (C0 and C1 are control characters and G0 and G1 are graphic characters), and escape sequences are used to select the C or G set for bytes with or without the high bit set. Graphic string does not allow control characters and General string does allow control characters.
You probably do not need all control characters; your schema should probably restrict which control characters are allowed in each context (although the ASN.1 schema format does not seem to have any way to do this). This way, you will only handle the control characters which are appropriate for your use.
This is messy, although canonical form simplifies it by adding some restrictions (this is one of the reasons why DER is better than BER, in my opinion). TRON code is better and is much simpler than the working of ISO 2022. (Unicode has a different kind of mess; although decoding is simpler, actually handling the decoded characters in text is its own big mess for many reasons. Unicode is a stateful character set, even though the encoding is stateless; TRON code is the other way around (and with a significantly simpler stateful encoding than ISO 2022).)
> the ASN1 docs discourage using the UniversalString and GraphicString types
UniversalString is UTF-32BE and GraphicString is ISO 2022 without control characters. By knowing what they are, you should know in which circumstances they should be considered useful or not useful; I think that they should not be discouraged in general (although usually if you want Unicode, you would use UTF-8 rather than UTF-32, there are some circumstances where you might want to use UTF-32, such as if the data or program is already UTF-32 for other reasons).
(The data type which probably should be avoided is the UTC time type, which is not Y2K compliant.)
> All these different string types are intimidating if I just want unicode/ascii
If you only want ASCII, use the IA5 type (or Visible if you do not want control characters); if you only want Unicode, use the UTF-8 string type (or Universal if you want UTF-32 instead for some reason). ("IA5" is another name for ASCII that as far as I can tell hardly anyone other than ITU uses.)
However, Unicode is not a very good character set, and they should not force or expect you to use it.
As I had mentioned before, you do not need to use or implement all of the ASN.1 data types; only use the ones appropriate for your application (so, if you do not like most of the types, then don't use those types). I also made up some additional nonstandard ASN.1 types (called ASN.1X), which also might be useful for some applications; you are not required to use or implement these either.
Unicode is an excellent character set, and for 99% of cases (much more probably) it's absolutely the best choice. So one should choose Unicode (and UTF-8) in all cases unless there is an excellent reason to do otherwise. As time passes there will be fewer and fewer cases where Unicode is not sufficient, so really we are asymptotically approaching the point at which Unicode is the only good choice to make.
This is all independent of ASN.1. But it is true that ASN.1 has built-in types for Unicode and non-Unicode strings that many other protocols lack.
Have you written up anything about ASN.1X anywhere? I'd love to take a look.
ASN1_BCD_STRING (64): Represents a string with the following characters: "0123456789*#+-. " (excluding the quotation marks). Each octet encodes two characters, where the high nybble corresponds to the first character and the low nybble corresponds to the second character.
ASN1_PC_STRING (65): Represents a string of characters in the PC character set. Note that the control characters can also be used as graphic characters.
ASN1_TRON_STRING (66): Represents a string of characters in the TRON character set, encoded as TRON-8.
ASN1_KEY_VALUE_LIST (67): Represents a set of keys (with no duplicate keys) and with a value associated with each key. The encoding is the same as for a SET of the keys, but with the corresponding value immediately after each key (when they are sorted, only the keys are sorted and the values are kept with the corresponding keys).
ASN1_UTC_TIMESTAMP (68): Represents a number of UTC seconds (and optionally fractions of seconds), excluding leap seconds, relative to epoch.
ASN1_SI_TIMESTAMP (69): Represents a number of SI seconds (and optionally fractions of seconds), including leap seconds, relative to epoch.
ASN1_UTC_TIME_INTERVAL (70): Represents a time interval as a number of UTC seconds. The number of seconds does not include leap seconds.
ASN1_SI_TIME_INTERVAL (71): Represents a time interval as a number of SI seconds (which may include fractions).
ASN1_OUT_OF_BAND (72): This type is not for use for general-purpose data. It represents something which is transmitted out of band (e.g. a file descriptor) with whatever transport mechanism is being used. The transport mechanism defines how a value of this type is supposed to be encoded with whatever ASN.1 encoding is being used.
ASN1_MORSE_STRING (73): Represents a string of characters in the Morse character set. The encoding is like a relative object identifier, where 0 means an empty space, and others is like bijective base 2 with 1 for dots and 2 for dashes, with the high bit for the first dot/dash, e.g. 4 means A and 8 means U.
ASN1_REFERENCE (74): A reference to another node within the same file. (Not all implementations will support this feature.) The encoding is like a Relative Object Identifier; the first number is how many times to go to the parent node (where 0 means the reference itself), and then the rest of the numbers specify which child node of the current node to go to where 0 means the first child, 1 means the second child, etc. It can reference a primitive or constructed node of a BER file, but you cannot specify a child index for a child of a primitive node, since primitive nodes cannot have child nodes. At least one number (how many levels of parents) is required, but any number of numbers is potentially possible.
ASN1_IDENTIFIED_DATA (75): Data which has a format and/or meaning which is identified within the data. The encoding is always constructed and consists of two or three items. The first item is a set of object identifiers, object descriptors (used only for display), and/or sequences where the first item of the sequence is a object identifier. The receiver ignores any items in this set that it does not understand. The second item in a ASN1_IDENTIFIED_DATA can be any single item of any type; it is interpreted according to the object identifiers in the first set that the receiver understands. The third item is optional, and if it is present it is a key/value list of extensions; the keys are object identifiers and the values are of any type according to the object identifiers. The default value of this key/value list is an empty key/value list.
ASN1_RATIONAL (76): Stored as constructed, containing two integers, being the numerator and the denominator. The denominator must be greater than zero. If it is canonical form, then it must be lowest terms.
ASN1_TRANSLATION_LIST (77): A key/value list where the keys identify languages. If the key is null then it means the default in case no language present in this list is applicable. The types of the values depends on the application (usually they will be some kind of character strings).
In addition, the same number for the BMP string type can also be used for a UTF-16 string type, and there is a "OBJECT IDENTIFIER RELATIVE TO" type which encodes a OID as either relative or absolute (in canonical form, it is always relative when possible) in order to save space; the schema will specify what it is relative to. ANY and ANY DEFINED BY are allowed despite being removed from the most recent versions of standard ASN.1. (The schema format for these extensions is not defined, since I am not using the ASN.1 schema format; however, someone who does use it might do so if they need it.)
There is also SDER, which is a superset of DER but a subset of BER, in case you do not want the mess of BER but do not want to require strictly canonical form either; and also SDSER which uses the same encoding for types and values than SDER but but length works differently in order to support streaming better.
As is usual, you do not have to use any or all of these types, but someone might find them useful for some uses. I have used some of them in my own stuff.
ValueList ::= SET OF KeyValue
KeyValue ::= SEQUENCE { key MyKeyType, value MyValueType }
MyKeyType ::= ... -- whatever you want here
MyValueType ::= ... -- ditto
?Your time types can be just an INTEGER with a constraint attached... (In Heimdal we use INTEGER constraints to pick a representation in the programming language.) E.g.,
-- 64-bit signed count of seconds where 0 is the Unix epoch
ASN1_UTC_TIMESTAMP ::= INTEGER (-18446744073709551616..18446744073709551615)
Out-of-Band ::= [APPLICATION 100] NULL
ASN1_REFERENCE is... interesting. I've not needed it, but some RPC protocols support intra-payload and even circular references, so if you have a need for that (hopefully you don't), then your ASN1_REFERENCE would be useful indeed.
ASN1_IDENTIFIED_DATA... ASN.1 has EMBEDDED-PDV, open types, and the TYPE-IDENTIFIER class -- there are many ways to do this in ASN.1. See https://github.com/heimdal/heimdal/blob/master/lib/asn1/READ...
ASN1_RATIONAL is just a tagged sequence of numerator and denominator, with a constraint that the denominator must not be zero.
OBJECT IDENTIFIER RELATIVE TO is just a CHOICE of OBJECT IDENTIFIER and RELATIVE IDENTIFIER.
Re: SDER... yeah, so Heimdal's codec produces DER but accepts a subset of BER for interop with OpenSSL and others. If you really want streaming then you'll want a variant of OER with fixed-length lengths (which IMO OER should have had, dammit), which then looks a lot like XDR but with different alignment and more types.
I had kind of expected a subset of x.680.
The abstract meaning matches, but the format is differently.
> ASN1_OUT_OF_BAND can just be a NULL with an APPLICATION tag or whatever
There are some uses of having a dedicated "out of band" type, such as being able to find them regardless of the schema (e.g. it might be used by a protocol that can use data with any schemas, but allows out of band data with any schema for some reason, and might want to modify the representations of out of band data when sending it to someone else).
> ASN1_IDENTIFIED_DATA... ASN.1 has EMBEDDED-PDV, open types, and the TYPE-IDENTIFIER class -- there are many ways to do this in ASN.1
EMBEDDED-PDV and those other things are different situations than I am doing, although it is similar, the use is not quite the same. ASN1_IDENTIFIED_DATA is simpler in some ways but also allows some things that EMBEDDED-PDV does not do.
Programs can also use ASN1_IDENTIFIED_DATA to identify the schema of a file that uses this type (and potentially be able to e.g. uncompress or decrypt it; this is the reason why the identifiers are allowed to be sequences and not only plain OIDs), or a part of another file.
> ASN1_REFERENCE is... interesting. I've not needed it, but some RPC protocols support intra-payload and even circular references, so if you have a need for that (hopefully you don't), then your ASN1_REFERENCE would be useful indeed.
Yes, it is what I thought too. So far I have not needed it either, but it might sometimes be useful.
> ASN1_RATIONAL is just a tagged sequence of numerator and denominator, with a constraint...
It can be defined as such in standard ASN.1, and the format is the same as that, but the abstract meaning is different. There is also a further constraint for the canonical form.
(Currently, the only place I have used this type is the tempo ratio in the .BGM lumps in Super ZZ Zero, but it would have other uses too, such as when converting data from other formats that have a rational number type.)
> OBJECT IDENTIFIER RELATIVE TO is just a CHOICE of OBJECT IDENTIFIER and RELATIVE IDENTIFIER.
It can be implemented that way in standard ASN.1 and has the same DER representation as your described type, although the abstract meaning is essentially the same as OBJECT IDENTIFIER and there is an additional constraint in the canonical form (as far as I know, this additional constraint cannot be written in standard ASN.1, but Super ZZ Zero cares about it being in canonical form (except for sound card identifiers in .BGM lumps, but this is an implementation detail for that specific part of the program)).
> I had kind of expected a subset of x.680.
Currently I am not using the schema format for ASN.1X (nor do I use the schema format of standard ASN.1); if someone else does then they might implement a variant of X.680 for use with ASN.1X. I probably would remove some stuff (and add some stuff) if I did make a variant, though.
(The use of ASN.1X is also not defined for JER, XER, OER, etc; if someone needs to, then they might do that.)
Yeah. I do remember a lot of workloads at Google where most of the CPU time was spent serializing/deserializing protos.
Not much is faster than protobuf except for zero copy formats.
- You can support all those features, and your ASM.1 library will be horribly bloated and over-engineered.
- You can support your favorite subset, but then you cannot say it's ASN.1 anymore. It will be "ASN.brabel", which only has one implementation (yours). And who wants that?
(unless you are Google and have immense developer influence... But in this case, why not design things from scratch, since we are making all-new protocol anyway?)
ASCII with escapes and UCS-2.
> horribly bloated and over-engineered.
It's no more or less complicated than XML, JSON or CSV. Which is why you can use ASN.1 to serialize to and from all these formats. ASN.1 provides you an additional level of schema above these. It simply allows you to describe your problem.
I find ASN.1 far more sane and useful than something like JSON Schema which is just as "bloated and over-engineered." It turns out describing data is not a simple problem.
What, your implementation does not include CP1036 to Unicode translation table? Sorry, it's no longer ASN.1, it's now ASN.themafia.
Oh, it does? Then how about xmlhstring vs hstring, are you handing difference properly?
What about REAL type? Does your asn.1 library include support for arbitrary-precision floating point numbers, both base 2 and 10? And no, you cannot use "double", I am sure there is an application out there which uses base 10 reals, or has 128 bits of mantissa.
ASN.1 is full of overengineered, ancient things like those, and the worst part - once you actually start using it to interoperate with other software, there is a good chance you'll see them. If you want something that people actually implement fully, choose something else.
Yes, read the standard, it's ASCII with special escape sequences. Which I don't have to render, I only have to convey them correctly across the network.
> What, your implementation does not include CP1036 to Unicode translation table? Sorry, it's no longer ASN.1, it's now ASN.themafia.
Why would I need the table?
> Oh, it does? Then how about xmlhstring vs hstring, are you handing difference properly?
What exactly needs to be "handled?"
> Does your asn.1 library include support for arbitrary-precision floating point numbers
Yes, because there are third party libraries which supply this functionality, so it's hardly any special effort to implement.
> ancient things like those
So no one ever needs arbitrary precision integers? You'll eventually need them for some application. Now all you have is ad-hoc implementations and almost no software to interoperate with them or verify them.
> If you want something that people actually implement fully, choose something else.
Name anything else with the same features yet is easier to implement "fully." Seriously, go read the JSON Schema specification. This is a _hard_ problem. If you think you've found an easy solution it's likely you've just left most of the functionality on the floor. And now we have to ask "is software X compatible with software Y?" Obviating the entire point of a standard.
The problem with ASN.1 -- the only real problem with ASN.1, is lack of excellent tooling.
Totally mad.
Technically, it sounds really good but the actual act of managing it is hell. That or I need a lot of practice to use them, at that point shouldn't I just use JSON and get on with my life.
Making changes to messages in a backwards-compatible way can be annoying, but JSON allowing you to shoot yourself in the foot will take more time and effort to fix when it's corrupting data in prod than protobuf giving you a compile error would.
If they live in their own project, making a single project be buildable with a git clone gets progressively more complex.
You now need sub modules to pull in your protobuf definitions.
You now also need the protobuf tool chain to be available in your environment you just cloned to. If that environment has the wrong version the build fails, it starts to get frustrating pretty fast.
Compare that to json, yes I don't get versioning and a bunch of other fancy features but... I get to finish my work, build and test pretty quickly.
Whether the team saves times in the longer when using protos is a question in its own.
It's not some perfect format.
That said, I'm disappointed with Protobuf too. Especially the handling of optional/default fields. I believe they did eventually add an `optional` tag so you can at least distinguish missing vs default field values.
The lack of required fields makes it very annoying to work with though. And no, there's no issue with required fields in general. The only reason it doesn't have them is because the implementation in Protobuf 2 caused issues and Google had to have a smooth transition from that.
If you're starting a greenfield project it seems silly to opt in to Google's tech debt.
If you're thinking "but schema evolution!!" well yeah all you need is a way to have versioned structs and then you can mark fields as being required for ranges of versions. So you can still remove required fields without breaking backwards compatibility.
Supports 64 bit ints, raw byte datatype, zero-copy parsing, does not require schema and can be converted to JSON for readability while retaining all field names.
It appears possible in some cases but it's not universally the case. Which means that similar binary transport formats that do support zero-copy, like Cap'n Proto, offer most or all of the perks described in this post, with the addition of ensuring that serialization and deserialization are not a bottleneck when passing data between processes.
"Cap’n Proto is INFINITY TIMES faster than Protocol Buffers. (...) there is no encoding/decoding step. The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out".
I take it as a rationalization of what OLE Compound File Binary - internal Microsoft Office memory structures serialized "raw" as file format - would look like if they paid more attention to being backward and forward compatible and extensible.
To actually literally avoid any copying, you'd have to directly use the messages in their on-the-wire format as your in-memory data representation. If you have to read them many times, the extra cost of dynamic getters can add up (the format may cost you extra pointer chasing, unnecessary dynamic offsets, redundant validation checks and conditional fallbacks for defaults, even if the wire format is relatively static and uncompressed). It can also be limiting, especially if you need to mutate variable-length data (it's easy to serialize when only appending).
In practice, you'll probably copy data once from your preferred in-memory data structures to the messages when constructing them. When you need to read messages multiple times at the receiving end, or merge with some other data, you'll probably copy them into dedicated native data structs too.
If you change the problem from zero-copy to one-copy, it opens up many other possibilities for optimization of (de)serialization, and doesn't keep your program tightly coupled to the serialization framework.
native > pb binary > native
vs
native > capnp binary > native
If you benchmark this, the two formats are very close. Exact perf depends on payload. Additionally, one could write their own protobuf serializer with protoc they really need to.
So my question is, why didn't Google just provide that as a library? The setup wasn't hard but wasn't trivial either, and had several "wrong" ways to set up the proto side. They also bait most people with gRPC, which is its own separate annoying thing that requires HTTP/2, which even Google's own cloud products don't support well (e.g App Engine).
P.S. Text proto is also the best static config language. More readable than JSON, less error-prone than YAML, more structure than both.
the twirp spec however is so simple you can throw together your own code generator in an afternoon for whatever language you want.
> → About 23 bytes.
This appears to be a nice ai-generated result. There are already at least 23 bytes of data there (number 42 (1 byte) + string of 5 chars + string of 17 chars + 1 boolean), so that plus field overhead will be more.
1. Size, biggest problem with JSON can happen when things gets too big. So here other formats might be better. Yet, as a reminder JSON has the binary version named BSON.
2. Zero batteries. JSON is readable by humans but also almost self explanatory format. Most languages has built in or quick drop in for json. Still, it’s easy to implement a limited JSON parser from scratch when in need (eg. Pure on func in C on a tiny device).
Working with Protobuf and MsgPack in the past, You have much more tooling involved especially if data passes between parts written in different languages.
3. Validation, JSON is simple. But there are solutions such as JSON Schema.
It is a JSON superset with binary encoding, created by MongoDB. (And there is even a JSON encoding of BSON, called extended JSON.)
AFAIK there is no widely adopted binary pure adaptation of JSON. (There are application-specific storage formats, like PostgreSQL JSONB, or SQLite JSONB.)
——-
Moreover, JSON is relatively compact. BSON or other self descriptive binary formats are often around the same size of JSON. MessagePack aggressively tries to be compact and is, depending on the data. BSON doesn’t try to be compact, rather it improves the parse speed.
However, yes, JSON compresses extremely well even in ancient gzip, but especially in Brotli, and desiring compaction of your API responses alone isn't necessarily the best reason to prefer a binary encoding of JSON to just using JSON and letting compression do its thing.
Correct. MessagePack comes out significantly ahead for small messages, otherwise it's nearly a wash.
> CBOR is probably the closest to a widely adopted "pure" adaptation of JSON
It adds (1) binary data (2) numeric variants including big integers (3) timestamps (4) tagged/semantic types.
I.e. it adds quite a bit. That's not necessarily a bad thing. Browser implementation would be great, but I'm skeptical.
Plus - tooling.
JSON might not be simpler or strict but it gets the job done.
If JSON's size or performance is causing you to go out of business, you surely have bigger problems than JSON.
Yes, there are abstractions or other hacks that would bolt on protobuf support in the browser, but that's not ideal in my mind.
Protobuf is an ideal exchange format when you're not dealing with the public as a whole. That is, a private or corporate API for your data processing pipelines, etc.
138 million downloads from npm in the last week. Yes, you can validate your JSON
The app 20req/sec
The app after optimizations: 20req/sec (It waits for db query anyway)
Otherwise, json is sufficient.
As an aside, like all things Google, their C++ library is massive (14mb dll) and painful to build (takes nearly 10 minutes on my laptop).
It's lead to a huge layer of adapters and native c++ classes equivalent to the protobuffers classes to try and mitigate these issues.
Due to very native nature of JSON in browsers and node backend it usually also the fastest data format
If for example you have C++ on backend and C++ on frontend - you'll definitely have some performance boost
But for browsers usage the goal is not so obvious
When you'll go through your own journey, and inevitably end back with json, do write another blog post :) ... we've all been there.
Distributing the contract with lock-step deploys between services is a bit much. JSON parsing time and size are not usually key factors in my projects. Protobuf doesn’t pose strict requirements for data validation, so I have to do that anyway. Losing data readability in transit is a huge problem.
Protobuf seems like it would be amazing in situations where data is illegible and tightly coupled such as embedded CAN or radio.
If you use good tooling, you can have a mutation change a variable type in the database and that type change is automatically reflected in the middleware/backend and the typescript UI code. Not only that libraries like HotChocolate for asp.net come with built-in functions for filtering, pagination, streaming etc.
We might disagree on what "efficient" means. OP is focusing on computer efficiency, where as you'll see, I tend to optimize for human efficiency (and, let's be clear, JSON is efficient _enough_ for 99% of computer cases).
I think the "human readable" part is often an overlooked pro by hardcore protobuf fans. One of my fundamental philosophies of engineering historically has been "clarity over cleverness." Perhaps the corollary to this is "...and simplicity over complexity." And I think protobuf, generally speaking, falls in the cleverness part, and certainly into the complexity part (with regards to dependencies).
JSON, on the other hand, is ubiquitous, human readable (clear), and simple (little-to-no dependencies).
I've found in my career that there's tremendous value in not needing to execute code to see what a payload contains. I've seen a lot of engineers (including myself, once upon a time!) take shortcuts like using bitwise values and protobufs and things like that to make things faster or to be clever or whatever. And then I've seen those same engineers, or perhaps their successors, find great difficulty in navigating years-old protobufs, when a JSON payload is immediately clear and understandable to any human, technical or not, upon a glance.
I write MUDs for fun, and one of the things that older MUD codebases do is that they use bit flags to compress a lot of information into a tiny integer. To know what conditions a player has (hunger, thirst, cursed, etc), you do some bit manipulation and you wind up with something like 31 that represents the player being thirsty (1), hungry (2), cursed (4), with haste (8), and with shield (16). Which is great, if you're optimizing for integer compression, but it's really bad when you want a human to look at it. You have to do a bunch of math to sort of de-compress that integer into something meaningful for humans.
Similarly with protobuf, I find that it usually optimizes for the wrong thing. To be clear, one of my other fundamental philosophies about engineering is that performance is king and that you should try to make things fast, but there are certainly diminishing returns, especially in codebases where humans interact frequently with the data. Protobufs make things fast at a cost, and that cost is typically clarity and human readability. Versioning also creates more friction. I've seen teams spend an inordinate amount of effort trying to ensure that both the producer and consumer are using the same versions.
This is not to say that protobufs are useless. It's great for enforcing API contracts at the code level, and it provides those speed improvements OP mentions. There are certain high-throughput use-cases where this complexity and relative opaqueness is not only an acceptable trade off, but the right one to make. But I've found that it's not particularly common, and people reaching for protobufs are often optimizing for the wrong things. Again, clarity over cleverness and simplicity over complexity.
I know one of the arguments is "it's better for situations where you control both sides," but if you're in any kind of team with more than a couple of engineers, this stops being true. Even if your internal API is controlled by "us," that "us" can sometimes span 100+ engineers, and you might as well consider it a public API.
I'm not a protobuf hater, I just think that the vast majority of engineers would go through their careers without ever touching protobufs, never miss it, never need it, and never find themselves where eking out that extra performance is truly worth the hassle.
In terms of human effort, a strongly typed schema rather than one where you have to sanity check everything saves far more time in the long run.
Also the “us” is ever-changing in a large enough system. There are always people joining and leaving the team. Always, many people are approximately new, and JSON lets them discover more easily.
It's the native format Erlang nodes use to serialize data to communicate with each other, but it's just a simple Type-Length-Value binary format, so anything can implement it.
It's small enough that you can create a reasonably complete and fast implementation of it for your language in an afternoon. It's self-describing, so if you can read ETF you can read any ETF message, as there are no out of band schemas. I love it.
[0] - https://www.erlang.org/doc/apps/erts/erl_ext_dist.html
But we have built protobuf into a web server that handles 2 requests per second. Why? We wanted to learn about it on the job.
I think that's 99% of Protobuf usage.
>If you develop or use an API, there’s a 99% chance it exchanges data encoded in JSON.
Just wondering if the inherent defiencies of JSON can somewhat be improved by CUE lang since the former is very much pervasive and the latter understand JSON [1],[2].
[1] Configure Unify Execute (CUE): Validate, define, and use dynamic and text‑based data:
[2] Cue – A language for defining, generating, and validating data:
Batching with message pooling to a transaction payload size limit actually made it performant. =3
If that is just your team, use whatever tech gets you there quick.
However, if you need to provide some guarantees to a second or third party with your API, embrace standards like JSON, even better, use content negotiation.
I too had this overly restrictive view of "APIs" for too long. One I started to think about it as the interface between any teo software components it really changed the way I did programming. In other words, a system itself is composed and that composition is done via APIs. There's no point treating "the API" as something special.
The design overhead involved in determining the correct URL and HTTP method adds a layer of subjectivity to the design and bike shedding arguments.
I’m not a huge fan of Protobuf/GRPC either, if there’s a better alternative I believe RPC is the right approach for exposing APIs.
I can't imagine using protobuf when you're in the first 5 years of a product.
Now, there is a serde_protobuf (I haven't used it) that I assume allows you to enforce nullability constraints but one of the article's points is that you can use the generated code directly and:
> No manual validation. No JSON parsing. No risk of type errors.
But this is not true—nullability errors are type errors. Manual validation is still required (except you should parse, not validate) to make sure that all of the fields your app expects are there in the response. And "manual" validation (again, parse don't validate) is not necessary with any good JSON parsing library, the library handles it.
Had the joy of implementing calling SOAP service with client generated from wsdl in .net core 2/3 times. Tooling was shit poorly undocumented amd with crappy errors at that time. Run only with very specific versions in very specific way. And not much you can do about it, rolling your own SOAP client would be too expensive for our team.
REST with JSON meanwhile is easy and we do it all the time, don't need any client, just give us request/response spec and any docs.
There is a really interesting discussion underneath of this as to the limitations of JSON along with potential alternatives, but I can't help but distrust this writing due to how much it sounds like an LLM.
Seems like the author just wanted to talk about Protobuf without bothering too much about the issues with JSON (though some are mentioned).
I promise you cannot tell LLM-generated content from non-LLM generated content. what you think you’re detecting is poor quality, which is orthogonal to the tooling used
I am not dismissing this as being slop and actually have no beef with using LLMs to write but yes, as you call out, I think it's just poorly written or perhaps I'm not the specific audience for this.
Sorry if this is bad energy, I appreciate the write up regardless.
I didn't find it confusing.
I found it unconvincing, but the argument itself was pretty clear. I just disagreed with it.
Getting everyone to agree on a standard was/is/will be the tougher part.
I was pushing at one point for us to have some code in our protobuf parsers that would essentially allow reading messages in either JSON or binary format, though to be fair there's some overhead that way by doing some kind of try/catch, but, for some use cases I think it's worth it...
A sizable portion of the integrations I've built have had to be built by hand, because there are inevitable stupid quirks and/or failures I've had to work around. For these usecases, using JSON is preferable, because it is easy for me to see what I have actually been sent, not what the partially up to date spec says I should've been sent.
This is consistent with the idea that communication over the internet should consist of (encrypted and compressed) plain text. It's because human beings are going to have to deal with human reality at the end of the day.
https://en.wikipedia.org/wiki/DCE/RPC
DCE/RPC worked in 1993, and still does today.
Protocol buffers is just another IDL.
Where!??