Comment by ltbarcly3 | Hacker News Reader

Comment by ltbarcly3 | Hacker News Reader

Watch as I don't use protobuf because it is horrible.

....

Tada!

If you patch clients to google services in Python to use json instead of grpc they get faster and more reliable. A lot faster. Benchmark it!

  def get_json_client() -> CloudLoggingQueryClient:
      """Client for log queries (JSON transport, avoids gRPC overhead)."""
      client = google.cloud.logging.Client.from_service_account_info(...)
      client._use_grpc = False
      return client

For me that is how I know something like protobuf is good. It is a nuisance to manage and distribute the definitions, adds a build step even to languages with no build step normally, is slower than almost every alternative, and artificially restricts you from doing lots of common things. It's so good!

And look at the code quality of the implementation! It's like a team of interns wrote it while drunk. It is a complete spaghetti mess, but has tons of super convoluted micro optimizations that are slower than just doing the most obvious thing, but make the implementation confusing and indirect. It's trash code.

What are good alternatives when you need a common "single source of truth" schema shared between multiple languages? We use protobuf between c# and Python.

The goal is not to have a single source of truth schema. That is a means to some other goal, and it's not even a good means.

If you never change the schema then you don't have to worry about it, get things working and never look back.

If you do change your schema from time to time, you need testing between the two systems. If you have good tests again a single source of truth is fully redundant, both systems are talking just fine. If you don't have tests things can and will break all the time even using protobuf.

> The goal is not to have a single source of truth schema. That is a means to some other goal, and it's not even a good means.

It’s about data transmission. Being able to encode and decode in a type safe manner between different languages (and so, different platforms) is a goal that makes a lot of sense.

> If you do change your schema from time to time, you need testing between the two systems

Or you could just use a defined format that doesn’t require testing. I rarely use protobuf but I can see why people do. The guaranteed backwards compatibility is huge for people who can’t just publish a new web frontend at the drop of a hat.

afavour it's been a long day already, maybe I can explain why you still need tests another day.

You could always try being less condescending when replying to users and you might find there was no need for anyone to reply in the first place.

If you understand how to evolve protobuf schema definitions, then you don’t really need testing. You instinctively know how the parser works when it is parsing data with a different schema from what it expects. And that’s a powerful thing. If your things break even when using protobuf then you don’t grok protobuf.

It’s probably not an exaggeration to say that being able to avoid tests between different systems who have different versions of the schema is a core goal of protobuf. Why? These two different systems are probably owned by different teams, and introducing explicit tests between different versions of them increases coupling between them.

"you don't really need testing"

Protobuf only allows you to add optional fields after the initial version, right? Because otherwise it would not be backwards compatible. Protobuf does not allow you to define contingent logic between fields. Optional fields are always nullable (or you must provide a default). This forces you to know all of this and use a method to see if it was actually set or just defaulted. So you have a ton of nullable/defaulted fields that likely are required to be filled in or not filled in, contingent on other values in the same struct. For example if the charge type is "purchase" then price must be not null and > 0. That sort of thing. This is so common you should just assume your app has a million little rules like this that are assumed. Protobuf does not help you here at all. You need some other logic to validate the data on top of protobuf. once you have that anyway protobuf's value is that it's expensive, requires build steps, is actually not fast, and forces you to distribute the schema between different apps somehow.

If it's not obvious yet, lets say you are sending your charge structs and you have a bug where sometimes you don't set price. It's defaulted to 0 or -1 or whatever nonsense value the default is, or null, it doesn't matter it's not correct sometimes. That is why you need tests my guy. Protobuf can't fix this. If you use json the tests make sure everything protobuf does for you is done too. In a world where you have to write tests because protobuf can't force you to correctly set fields, you have tests already, and protobuf didn't help you at all. Whether you call it tests or input validation or whatever, protobuf definitions are insufficient, and when you have what is sufficient it 100% covers everything protobuf does 'for you'.

If you don't get it at this point then lets just agree that you will never get it.

> Protobuf only allows you to add optional fields after the initial version, right?

That’s not all. For example, you can also change fields from optional to repeated or vice versa. For another example, you can also delete fields.

> Protobuf does not allow you to define contingent logic between fields.

You are just saying that protobuf is not a data validation library. That’s true; it only handles data serialization. You need to write validations yourself. And you will most likely need tests to test your validation code. But those are entirely tests within a single process; they are not tests involving a producer and a consumer of a protobuf message, which is the wrong kind of tests.

> For example if the charge type is "purchase" then price must be not null and > 0. That sort of thing.

That sort of thing sounds like you are not using `oneof` appropriately. The charge type shouldn’t be a field. It should be a submessage called Purchase.

Both sibling comments say one type of assurance makes the other irrelevant, but I would wager they cover different territory.

Json schema?

I quite like the look of Typespec though I haven't used it much.

I always thought Thrift was waaay better than any of the alternatives, but it always had terrible documentation and I think it died mainly because of that.

Is that a recent-ish improvement? I feel like HTTP/2 would be roughly the same performance for JSON and protobuf, so maybe this is HTTP/2 vs HTTP/3?

I think the overhead is protobuf itself but I can't check.

Protobuf simply encode things way more efficient that JSON can define a single object. You're quite frankly spewing bullshit in this whole thread.

You don’t get what they say. It’s not about about how efficient it is after encode, it’s about how fast encode is. They are not spewing bs, they’re focusing on a single point. The question is: do you send it over the wite more often than performing encode/decode.

That's what the person you replied to is talking about, and they're right. Putting aside the final byte size (where protobuf also wins), protobuf is faster at both encoding and decoding than json. There are numerous benchmarks you can find that show this.

The advantages of json are not related to performance.

doesn't seem universally true, https://github.com/protobufjs/protobuf.js/issues/2114#issue-...

If you're writing JS you cannot beat JSON.parse, because you're running the most optimized C++ implementation of JSON which will outcompete any decoder written JS itself.

Which is not a very generalizable situation.

Ok but that's the point. You are just explaining why the 'protobuf is faster' claim is generally false.

It is generally correct. When both implementations are completely in the same language Protobuf will win. Instead of a pure-JS implementation one can also make a FFI module for Protobuf.

The comparison that's being made is equivalent to implementing quick sort in Python and bubble sort in C++ and then declaring bubble sort is the better alternative.

Anyone with a healthy understading of computer science and programming experience will not make bullshit claims like that.

You say 'generally correct', but then to make it correct you have to make a bunch of specific constraints. That's not general. I'm comparing the FFI using library in python to the json library in python, and json is faster for me. I showed how in the post. You are welcome to find my mistake.

I was the grandparent reply and I meant to only carve out "you cannot write anything in JavaScript faster than anything written in tuned C, and json.parse is the latter".

Which link were you comparing FFI in python to json in python? The link I see seems to be about pure JavaScript language Protobuf impls vs json.parse?

No, protobuf is not faster, not on Python, because google's implementation is poor. Python is the #1 language in the world right now. Don't use protobuf.

That's not the lesson I'm taking away from this. Imagine how much faster the code would be if you didn't use Python.

Comparing a wrapped C++ gRPC backed stack with an httpx/requests backed one is like comparing apples to elephants.

Is this because load is lighter on their JSON endpoints that their gRPC ones?

so how do you save data over the cable when it's needed?

JSON. /s

zstd?? Obviously?

Eventually it’s all bytes. Where do you want to save them?

Reply on news.ycombinator.com