The reason to use Protobuf is not performance

Does protobuf or capnproto have good tooling support out in the world?

Proto was very convenient at Google since everything was proto, your debug tools were proto, all your data pipeline tools spoke proto, you had command line utilities for protos, etc, etc.

It seems like the rest of the world is on JSON unless they actually need performance, so if you don't get performance, should you use something like a code generator for JSON types like Conjure so that you don't actually have to deal with these binary formats everywhere?

I don't think a large group can pick a standard. What works well for one programming language does not work for another, and we devolve into tribes. Protobuf gained popularity when it dropped features to better support Go more naturally (no more null/nil for example), and rode in on a wave of Go adoption. And made things worse from a Python developer perspective. Pretty much the same as toml adoption, which fits more naturally with Go and more static languages, unlike the much more dynamic and flexible yaml.

GRPC <=> Rest transcoding works fine, you can do this with envoy for example.

And if you use the connect protocol (also from buf) then you can also use json.

So it's quite achievable to get all the benefits you want from the gRPC ecosystem for developing your service while also allowing quick and dirty client usage (i.e. scripting with cURL).

I fail to see how running something like protoc before running or compiling code is a show stopper. There are even npm packages that will install protoc for you [1]. That means you can use it in a custom npm build script action, or invoke it before staring the dev server? Put in a git pre-commit to run it to assert there are no changes? Run via a file-watcher?

IMO writing a schema in something like JSON for OpenAPI is way more verbose and time consuming than writing a proto file and a script or Makefile to invoke protoc. There also is the mismatch in capabilities. The JS-adjacent schema standards mostly do not directly support native types like uint64, let alone directly specify how to encode numbers like fixed64. I mean, look... the work around in JavaScript still is use "string" [2].

On the other hand, there is a standard in OpenAPI for validating values. But that doesn't make sense in the Protobuf world since usually the API is expected to be forward and backward compatible, within reason, so something like a min and max value needs to be done at runtime, and not fixed within the schema. And don't get me started on trying to define cross-field validation in a declarative schema. It is easier to just write the code. :-)

So, IMO, no, a single standard won't happen in the industry. The many dimensions like logical vs physical models, validation, or error handling that make it impossible.

[1] https://www.npmjs.com/package/@protobuf-ts/protoc [2] https://github.com/OAI/OpenAPI-Specification/issues/2617#iss...

Protobuf has a JSON canonicalization [0]. All our gRPC servers automatically accept HTTP/JSON traffic with the procedure name specified in a header. That's what I use when I'm writing a script.

https://protobuf.dev/programming-guides/proto3/#json

If only JSON-RPC[1] were more popular for JSON-based APIs. With JSON-RPC specifically, the trade-off is that it would not work well with binary data; but if that's a problem then instead of JSON it could be BSON, bencode, EBML, or whatever (at that point it wouldn't be JSON-RPC though), the point is that you can nest stuff without switching syntax.

But compared to HTTP, if what you want is just JSON, then JSON-RPC saves you from having 5 different places to pass data, each with its own special syntax (one for method name; one for path; one for query params in the path; one for headers; one for body with the actual data).

We only not notice this because of just how many libraries are doing the hard work for us and making it seem like HTTP requests are clean and nice stuff; but no, HTTP is certainly not a nice protocol. Its main advantage is its ubiquity.

People may ask, "what about caching and stuff?". Well, with HTTP we made that work by passing a special key in the part of the request that is delimited by CRLF and where each line contains a key-value pair separated by a colon-and-space; and the clients know that a response was cached because they can read the first line of the response, which is a space-separated list of keywords that has a specific numeric literal in its second position (yes, I'm oversimplifying).

So I'm pretty sure we can do something better with fewer different syntaxes. And reverse proxies would presumably have it harder to be vulnerable to things like request smuggling[2].

[1]: https://en.wikipedia.org/wiki/JSON-RPC [2]: https://en.wikipedia.org/wiki/HTTP_request_smuggling

> pick a standard (openapi? jsonschema? something else?)

This is a false dichotomy, as OpenAPI and JSON Schema are complementary: as of version 3.1, OpenAPI use JSON Schema to describe request and response payload content (prior to that version it was almost compatible).

They describe different things: OpenAPI describes the request-response API model, while JSON Schema can be used to specify contents of any JSON-encoded data -- for example in streaming messages or document databases.

There is also AsyncAPI, which is designed to document and specify the event-based API, for example in a system using Apache Kafka or a similar queue to communicate.

Until you have to put some binary data in the JSON of course, at which point most of the supposed benefits of JSON go right out the window.