LinkedIn adopts protocol buffers and reduces latency up to 60%

More seriously though, I wish we used generic terms instead of using brands. "Look up" (or lookup as a noun) or "search" can't be too hard to use. We don't need to turn our thoughts into ads at every corner.

(so, yes, I'll take your nope for my whynotting)

Now, if you are launching a search engine, you need to name it wisely for when people verb it. As a brand you probably want this to happen. Duck Duck Go was not very wise.

If you're only incrementing when you change a particular call, then you end up with /api/myservice/create/v2 (or sometimes this is done via header) but v5 for the update call, and understanding what version goes to what becomes a cognitive overhead.

(and really the problem isn't basic CRUD endpoints, it's the ones with complex logic and structure where what's being built isn't necessarily the same thing over time.)

It's one thing when v2 and v5 are the latest, but if someone else comes through later and wants to bolt on a feature to a service that is trying to talk to v2/v5 when v3/v9 are the latest, you have to go back and look up a map of which endpoints are contemporary if you want to add a third call (v2/v5/v2) that is supposed to work together.

This can be done via swagger/etc but you are essentially just rebuilding that service versioning with an opaque "api publish date" built over the top.

I was thinking along the same lines. It is easy to make it sound like you have a lot of endpoints, when the vast majority are likely API mappings pointing to the same underlying service.

API endpoints is is almost as weird of a metric as LoC. It does tell you something, but in a way that can be misleading.

Yes, this is what I meant. At least a service should be versioned as a single unit, it’s /api/myservice/v2/endpoint. But if you have 10 endpoints in your service and 10 versions, it’s still 10x10 even if most of them don’t change.

It would be a nightmare to consume something like /api/myservice/endpoint/v2. Needing v2 of the create endpoint but only v5 of the update? That would be ugly to try and work against. And actually there is no guarantee versions are even behavior compatible (although it would be stupid for it to wander too far). There can be cases where response objects don’t convey some info you need in some versions etc.

I was thinking of service as being the unit of "API" here rather than an API consisting of multiple services, "each service provides its own API" is how I was thinking of it. But I can see the usage of saying "this is our [overall] public/internal APIs" too. And I agree /api/v2/myservice would be a bit much if every service moved the global version counter every time a single endpoint was changed lol

(although I suppose you could make an argument for "timestamp" as a "point in time" API version, if you version the API globally. Sounds like it would cause friction as services try to roll out updates, but it's notionally possible at least.)

They're different, but not that different as the functional message-oriented nature of Erlang means your entire state is typically in transferable terms, and the serialization format directly maps terms back and forth.

Technically Erlang could go much further, but much like multicore support took forever, I guess due to lack of funding, it doesn't. Things like:

1. When transferring between two compatible nodes, or processes on the same node, 'virtually' serialize/deserialize skipping both operations and transferring pointer ownership to the other process instead.

2. When transferring between compatible nodes on different servers, use internal formats closer to mem representation rather than fully serializing to standard ETF/BERT

I rememeber being spammed by zynga games on behalf of my facebook contacts. So I never went near them, in browser or on mobile.

I can speak only from my experience. I was in a team doing ads evaluations, and we had multiple other teams specializing in various bits & pieces of the Ads infrastructure - for example one team would keep what new amenities were on a hotel/motel, etc. - so in order to fill all these mundane, but intricate details - e.g. "bool has_pool" of sorts, it'll keep adding fields, or introducing new proto messages to fully explain capture that data. And then each team would own several if not hundreths of these protos, these protos would be bundled as `oneof` (e.g. "union") or something like this in a bigger encapsulating structure - so in your

row database (bigtable), each team would have a column that stores their proto (or empty). So we'll run a batch process to read each such column, and then treat all the fields in the proto stored there as data, and expand all these fields as individual columns over the the columnar (read-only) db.

Later an analysts/statistician/linguist/etc. can query that columnar db with data they are interrested about. So that's what I remember (I left several years ago, so things might've changed), but pretty much instead of typical for row-databases have a column for everything - you just have a column for your protobuf (a bit like storing HSON/JSON in postgres), but then have the ETL process mow down through each field in that "column" and create "columns" for each such field.

We had to do some workarounds though as it was exporting too much, and it was not clear how we can know which fields would be asked about (actually there was a way probably, but it'll taken time to coordinate with some other team, so it was more (I think) on coordination with internal "customers" to disallow exporting these).

But the cool thing, is that if customer team added new field to their proto, our process would see it, and it'll get expanded. If they deprecate a proto, there could be (not sure if there was, but could be added) - no longer export it. But for this to work you need the "protodb" e.g. to introspect, able to reflect actual names in order to generate the column.

Most database schemas at Google have exactly two "columns": a primary key column, and an "Info" column which contains a large protobuf. The difference between that and a simple K/V store, though, is that the storage engine is building indices and columnar data stores under the hood that make it perform as if it had been defined with all the fields as flattened columns, but you never actually have to declare a schema in any detail. That cuts down dramatically on tooling and release nonsense - there's nothing like a Rails migration script, because the only update you're likely to do is "sync to latest proto definition". It also means that you have all the semantics of a protocol buffer to define your data structure - oneofs to express optionality, and submessages to group fields together - which simply don't exist in a normal DBMS.

They really are optional though. This is inherent in the problem if you want to support arbitrary past and future versions. You need to decide what to do when the data is missing. Open file formats are a difficult problem. [1]

I might define a different class for a validated object, so version skew only needs to be dealt with at the system edge.

Maybe you don’t actually have that problem because you can make a closed-world assumption? For example, you control all writers and aren’t saving any old-format data, like in a database that has a schema. In that case you can make fields required.

[1] https://acko.net/blog/on-variance-and-extensibility/