Hacker News Reader

Comment by parhamn | Hacker News Reader

It's interesting, to everyone but but the mega shops like Google, protobuf is a schema declaration tool. To the megashops its a performance tool.

For most of my projects, I use a web-framework I built on protobuf over the years but slowly got rid of a lot of the protobufy bits (besides the type + method declarations) and just switched to JSON as the wire format. http2, trailing headers, gigantic multi-MB files of getters, setters and embedded binary representations of the schemas, weird import behaviors, no wire error types, etc were too annoying.

Almost every project I've tracked that tries to solve the declarative schema problem seems to slowly die. Its a tough problem an opinionated one (what to do with enums? sum types? defaults? etc). Anyone know of any good ones that are chugging along? OpenAPI is too resty and JSONSchema doesn't seem to care about RPC.

View on HN

> It's interesting, to everyone but but the mega shops like Google, protobuf is a schema declaration tool

There are lots of other benefits for non performance-oriented teams and projects: the codegen makes it language independent and it's pretty handy that you can share a single data model across all layers of your system.

If you don't care about the wire format, the standard JSON representation makes it pair well with JSON native databases, so you can get strict schema management without the need need for any clunky ORM.

That's assuming JSON native databases are fine for your use case though, but in practice it's only good for storing documents that don't need to be edited/queried much by the backend storing them.

> in practice it's only good for storing documents that don't need to be edited/queried much by the backend storing them

Why aren't they good for that? They can have very high write throughput, they don't require ORMs, and they can be indexed and queried using standard database methods like the SQL language.

You can even enforce strict schemas on them if you want to, just as you would with an RDBMS.

At scale, the performance gains can be dramatic. For example, moving a json web service to CBOR, I was able to squeeze 15% more throughput out of existing hardware. When you’re dealing with hundreds, if not millions of requests per minute this can be financially prudent.

I am curious about this, whats the difference between the fastest json libraries (e.g simdjson) and protobuf?

In my case I was using CBOR, not protobuf. The effect could be similar though when moving from json to protobuf. You get a much more efficient serialization and deserialization. This reduces CPU demand on server side for sending response payload and for client to receive the payload.

From Amazon: https://smithy.io/2.0/index.html (internally known as Coral)

nit: Coral and smithy are not comparable.

Coral is a schema definition language, yes. But it’s also a full rpc ecosystem.

Smithy at this point is only really an IDL that (in most cases, at least before I left) is “only” used to generate Coral models and then transitively Coral clients and services. The _vast_ majority of Amazon is still on “native” Coral

> OpenAPI is too resty

I'm curious as to why would you think that?

There's a bit of boilerplate in there if you want to use it for a naive implementation, but I don't find it exceedingly resty.

From my POV, using protobuf as a schema declaration tool (as opposed to being a performance tool) is blind follower behaviour. Getting over all the hurdles doesn't seem worth it for the payoff, and it only becomes less valuable when compared to all the OpenAPI tooling you could be enabling instead.

This being for a web-based problem, where we're solving schema declaration.

> I'm curious as to why would you think that?

Huh? From the Swagger site:

"The OpenAPI Specification defines a standard interface to RESTful APIs..."

From their OpenAI Initiative:

"The OpenAPI Specifications provide a formal standard for describing HTTP APIs"

Not sure I care to understand your POV more given this obvious bit was missed by you.

Reply on news.ycombinator.com