What are good alternatives when you need a common "single source of truth" schema shared between multiple languages? We use protobuf between c# and Python.
I always thought Thrift was waaay better than any of the alternatives, but it always had terrible documentation and I think it died mainly because of that.
If you never change the schema then you don't have to worry about it, get things working and never look back.
If you do change your schema from time to time, you need testing between the two systems. If you have good tests again a single source of truth is fully redundant, both systems are talking just fine. If you don't have tests things can and will break all the time even using protobuf.
It’s about data transmission. Being able to encode and decode in a type safe manner between different languages (and so, different platforms) is a goal that makes a lot of sense.
> If you do change your schema from time to time, you need testing between the two systems
Or you could just use a defined format that doesn’t require testing. I rarely use protobuf but I can see why people do. The guaranteed backwards compatibility is huge for people who can’t just publish a new web frontend at the drop of a hat.
It’s probably not an exaggeration to say that being able to avoid tests between different systems who have different versions of the schema is a core goal of protobuf. Why? These two different systems are probably owned by different teams, and introducing explicit tests between different versions of them increases coupling between them.
Protobuf only allows you to add optional fields after the initial version, right? Because otherwise it would not be backwards compatible. Protobuf does not allow you to define contingent logic between fields. Optional fields are always nullable (or you must provide a default). This forces you to know all of this and use a method to see if it was actually set or just defaulted. So you have a ton of nullable/defaulted fields that likely are required to be filled in or not filled in, contingent on other values in the same struct. For example if the charge type is "purchase" then price must be not null and > 0. That sort of thing. This is so common you should just assume your app has a million little rules like this that are assumed. Protobuf does not help you here at all. You need some other logic to validate the data on top of protobuf. once you have that anyway protobuf's value is that it's expensive, requires build steps, is actually not fast, and forces you to distribute the schema between different apps somehow.
If it's not obvious yet, lets say you are sending your charge structs and you have a bug where sometimes you don't set price. It's defaulted to 0 or -1 or whatever nonsense value the default is, or null, it doesn't matter it's not correct sometimes. That is why you need tests my guy. Protobuf can't fix this. If you use json the tests make sure everything protobuf does for you is done too. In a world where you have to write tests because protobuf can't force you to correctly set fields, you have tests already, and protobuf didn't help you at all. Whether you call it tests or input validation or whatever, protobuf definitions are insufficient, and when you have what is sufficient it 100% covers everything protobuf does 'for you'.
If you don't get it at this point then lets just agree that you will never get it.
That’s not all. For example, you can also change fields from optional to repeated or vice versa. For another example, you can also delete fields.
> Protobuf does not allow you to define contingent logic between fields.
You are just saying that protobuf is not a data validation library. That’s true; it only handles data serialization. You need to write validations yourself. And you will most likely need tests to test your validation code. But those are entirely tests within a single process; they are not tests involving a producer and a consumer of a protobuf message, which is the wrong kind of tests.
> For example if the charge type is "purchase" then price must be not null and > 0. That sort of thing.
That sort of thing sounds like you are not using `oneof` appropriately. The charge type shouldn’t be a field. It should be a submessage called Purchase.