Hacker News Reader

Comment by tijsvd | Hacker News Reader

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively.

Google's --------------- 20.6% 26.5Ki 45.7% 12.1Ki 39.3% 10.4Ki 5.0% 1.33Ki 3.1% 2.4% 1.2% 1.2% 1.1% 1.1% 296 [1 Others] 19.1% 24.5Ki JSON 60.4% 14.8Ki 39.6% 9.67Ki 14.7% 18.9Ki 81.9% 15.4Ki upb/def.c 18.1% 3.41Ki 13.9% 17.8Ki Generated Code 46.9% 8.34Ki 17.6% 3.12Ki 16.4% 2.92Ki 14.7% 2.61Ki 2.0% 1.4% 0.5% 88 [2 Others] 0.3% 0.3% 10.0% 12.9Ki Binary Format 54.0% 6.95Ki 46.0% 5.92Ki 8.8% 11.3Ki Core Runtime 66.3% 7.52Ki 17.8% 2.02Ki upb/upb.c 15.9% 1.80Ki upb/msg.c 5.6% 7.19Ki [34 Others] 3.8% 4.94Ki Text Format 100.0% 4.94Ki 3.5% 4.46Ki 100.0% 128Ki TOTAL So JSON is pulling In these cases, implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at least some implementation available.

Then, the author laments the wire format itself for having varint-encoded length prefixes, which can not be fixed up. That is true, but it's not that much of a problem. Most straightforward is to simply go through nested data multiple times, once to calculate the length, and again for the actual encoding (and then again and again for deeper nesting).

What makes this bearable is the fact that data will be mostly loaded into L2 cache (L1 for smaller messages) on the first pass, which makes the next pass much faster.

The story breaks down for large, deeply nested messages, but then, the topic here is telemetry which I would expect to consist of a stream of small, shallow messages.

View on HN

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered).

Right, they were designed for use in Google's servers, where binary size is mostly irrelevant, while speed and features (e.g. reflection) are valued.

"Lite mode" (not mentioned in the article, for some reason) optimizes for code size instead. Admittedly, it's not as small as an implementation written from scratch with minimal code footprint as the main goal, but it does cut the size quite a bit...

(Disclaimer: I wrote those C++ and Java implementations, but that was 10-15 years ago, things may have changed in the meantime...)

HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply!

Lite mode is neat, I hadn’t seen that. That’s super useful for my use case, cheers. Any gotchas with it to be aware of?

> Any gotchas with it to be aware of?

Lots, unfortunately. E.g. you can't stringify objects for debug printing, because stringification is implemented in terms of reflection.

> HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply!

Not just that, I suspect it’s a sign of HN’s unique culture that the “smack talk” seems not to be taken personally and instead a highly nuanced and interesting discussion ensues!

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered).

As someone who has been working on protobuf-related things for >10 years, including creating a size-focused implementation (https://github.com/protocolbuffers/upb), and has been working on the protobuf team for >5 years, I have a few thoughts on this (thoughts are my own, and I don't speak for anybody else).

I think it is true that protobuf C++ could be a lot more lean than it currently is (I can't speak to Java as I don't work on it directly). That's why I created upb to begin with. But there's also a bit more to this story.

The protobuf core runtime is split into two parts, "lite" and "full". If you don't need reflection for your protos, it's better to use "lite" by using "option optimize_for = LITE_RUNTIME" in your .proto file (https://developers.google.com/protocol-buffers/docs/proto#op...). That will cut out a huge amount of code size from your binary. On the downside, you won't get functionality that requires reflection, such as text format, JSON, and DebugString().

Even the lite runtime can get "lighter" if you compile your binary to statically link the runtime and strip unused symbols with -ffunction-sections/-fdata-sections/--gc-sections flags. Some parts of the lite runtime are only needed in unusual situations, like ExtensionSet which is only used if your protos use proto2 extensions (https://developers.google.com/protocol-buffers/docs/proto#ex...). If you avoid these cases, the lite runtime is quite light.

However, there is also the issue of the generated code size. Unlike the runtime, this is not a fixed cost, but is proportional to the number of messages you use. If you have a lot of messages it can quickly dwarf the size of the runtime. For this reason, C++ also supports "option optimize_for = CODE_SIZE" which uses reflection-based algorithms for all parsing/serialization/etc instead of using generated code. This means you pay the fixed size hit from linking in the full runtime, but the generated code size is much smaller. On the downside, "optimize_for = CODE_SIZE" has a severe ~10x speed penalty for parsing and serialization.

I have long had the goal of making https://github.com/protocolbuffers/upb competitive with protobuf C++ in speed while achieving much smaller code size. With the benefit of 10 years of hindsight and many wrong turns, upb is beginning to meet and even surpass these goals. It is an order of magnitude smaller than protobuf C++, both in the core runtime and the generated code, and after some recent experiments it is beginning to significantly surpass it in speed also (I want to publish these results soon, but the code was merged in this PR: https://github.com/protocolbuffers/upb/pull/310).

upb has downsides that prevent it from being fully "user ready" yet: the API is still not 100% stable, there is no C++ API for the generated code yet (and C APIs for protobuf are relatively verbose and painful), it has a bunch of legacy APIs sitting around that I am just on the verge of being able to finally delete, and it doesn't support proto2 extensions yet. On the upside, upb is 100% conformant on every other protobuf feature, it supports reflection, JSON, and text format, but also lets you omit these if you don't want to pay the code size.

I hope 2021 is a year when I'll be able to publish more about these results, and when upb will be a more viable choice for users who want a smaller protobuf implementation.

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

[Also a googler]:

PB is used for a wide variety of things, its a lingua-franca of data interchange within Google, but web (frontend) still often uses json. So if I want to have some proto in a database and render it on a UI to a user, somewhere in there I'm probably going to be translating the raw proto to either json or json-like (I'll admit I'm not sure how stuff like grpc-web fits in here, if you can just get proto-serialized bytes passed around in some a blob, but there are some pretty low-overhead json proto encodings that exist anyway).

Sure that makes some sense.

But it's somewhat ugly as well, from an architecture point of view. Because that argument translates to anything else you might want to do with these objects.

In past libraries for C++ I've tried to prevent this kind of coupling by adding generated template methods like "walk(f)" where f would be a templated callable, called with a descriptor and data reference for each field. Any kind of pretty printer or SQL statement can be built that way.

People implementing public-facing APIs might not be comfortable forcing all the consumers of that API to use Protobuf, but may want to use Protobuf internally within their service. So, Protobuf lets you convert to/from JSON, so that your service can expose a JSON API while maintaining Protobuf schemas and never dealing with a JSON parser directly.

The implementation is based on Protobuf's existing reflection features, so if you are already compiling with that enabled, the JSON implementation does not add much code bloat on top of that.

> Don't people use pb so they do not have to deal with JSON?

I think that's one reason, but there are others too, like getting generated type-safe accessors and an explicit schema. I've sometimes used protobuf for this reason even when I'm not planning to use binary format.

> I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

Here is a breakdown of a binary that uses upb and links in binary format, JSON format, text format, and some generated protos:

         VM SIZE Generated Reflection bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upbdefs.c bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upbdefs.c bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upbdefs.c 832    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upbdefs.c 648    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upbdefs.c 320    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/timestamp.upbdefs.c 312    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upbdefs.c 304    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/field_mask.upbdefs.c upb/json_decode.c upb/json_encode.c Reflection Runtime upb/reflection.c bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upb.c bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upb.c bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upb.c bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/descriptor.upb.c 356    bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upb.c 264    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upb.c 48    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/any.upb.c 48    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upb.c upb/decode.c upb/encode.c upb/table.c upb/text_encode.c [section .rodata] in 25Ki directly, plus it depends on the 19Ki reflection runtime and the generated reflection.But it is all a la carte: if you don't want JSON, just don't use it and don't link it in.
(btw that 7Ki for upb/table.c is to support protobuf maps, I want to make that a la carte also so that you don't pay for it when you don't use maps).

How'd you generate that listing? It seems useful.

I used my tool Bloaty, which parses the ELF and DWARF information in the binary: https://github.com/google/bloaty

Thanks!

Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated?[edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?

Probably a complaint of the codegen and it’s wacky name mangling.

Probably all this "security" nonsense. What could possibly go wrong with implementing a binary serialisation protocol your self...

Oh look sarcasm! On Hacker News!

Things may have improved since, but the implementations are somehow very large and slow.Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an encode/decode pair would take over a microsecond with Google's implementation, but about 50 ns with a simpler one (versus 15 ns for a memcpy).
For Java my experience is mostly with the API itself, which felt very heavy.
Edit: I think a lot depends on your use case. I use protobuf mostly as a 'trusted' protocol. If someone didn't set a required field, I don't care. Some bloat may have to do with verifications that I've never needed.

> Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messagesThis is no longer the case if you use arenas: https://developers.google.com/protocol-buffers/docs/referenc...
> and perhaps even for optional scalars
This has never been the case, except for string fields where std::string forces us to allocate.
Ideally we will eventually use std::string_view for string accessors instead of std::string, so that even string data can be allocated on an arena instead of the heap.

> This has never been the case, except for string fields where std::string forces us to allocate.I'm was quite surprised you didnt offer your own stringview implementation (or something similar) the last time I looked at protobuf. I'd naively assume that inside Google this could be quite a low-efford high-reward optimization.

> I'm was quite surprised you didnt offer your own stringview implementation (or something similar) the last time I looked at protobuf.We sort of do actually: https://github.com/protocolbuffers/protobuf/blob/master/src/...
The internal version of protobuf lets you switch individual string fields to string_view using [ctype=STRING_PIECE], but migrating the default away from std::string is mainly just an enormous migration challenge.
Internally we also do something slightly nuts: we break the encapsulation of std::string so that we can point it to arena-allocated memory (we then "steal" the memory back before the destructor runs).  We can only afford to do this internally, where the implementation of std::string is known. The real long-term solution is to move to string_view.

It was never really the case. It's just before arenas it depended on your underlying heap allocator to do all the hard work.Arenas have been around for a long while now though...

> AFAIK the C++ implementation would always allocate on the heap for nested messagesFWIW if you reuse the same message object for multiple parsings, it will re-use the sub-objects as well, thus amortizing away the allocation cost. Parsing the same message into the same object twice should do zero allocations on the second parse. This is the intended way to use Protobuf for small-size messages.
Apparently the C++ implementation has also grown support for arena allocation more recently. (After my time, so I don't know much about it.)

But then all that must come with bookkeeping, which brings its own cost.Take a look at an implementation like Prost, for Rust. It's very similar to what I did (10 years ago by now). Everything is just inline, except when messages can be recursive (which should be rare for most protocols).

The bookkeeping is not that hard... the pointer is null until it is first allocated, then it remains non-null, while a separate boolean indicates whether the sub-message is actually present in the parent.The big problem is bloat in memory usage if you parse many differently-shaped messages, requiring the app to implement hacks like only reusing a particular object a certain number of times.

> Everything is just inline, except when messages can be recursive (which should be rare for most protocols).Many messages are have lots of optional sub-message fields, and set only a few of them in any given message.  These messages would be huge if everything is inline (especially if the same thing happens in those sub-messages).
I agree that inlining all sub-messages works great for dense schemas, but it assumes too much about the schema to be a good design for a general-purpose proto library I think.  Also maps and repeated fields can never be inline.

Yes, this comes back to use case.So it's great that there are different implementations for different use cases. It helps that the wire format is simple and well-documented.

I wish I had known this a few years ago, I could have saved some allocations. :)

Since all other comments appear to contradict you and make apologies from authority (after all, Google can't do anything wrong, right?) I'd like to just reassure you with my 20 years of experience developing C (and the last 10 with C++) in network and system software: that any library -- any library -- that forces internal dynamic memory upon its user smells bad. It's not a universal condemnation, but it begs the question, or rather the skepticism to ask why?In the case of protobufs, there is no real answer. Protobufs is one of the few network libraries that doesn't operate with zero-copy. That steps over a tangible line in the sand.
No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications.

> No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications.That's a bit harsh. Protobufs deliver smaller wire size than any of the newer "zero-copy" formats. And many receivers of zero-copy formats will... copy the data into some internal representation. If your protobuf implementation delivers classes that are good enough to work with internally (store in maps, forward, etc) then you don't really lose something; instead you gain, due to no manual conversion layer.

CBOR also delivers optimal wire sizes with its variable length encoding. Ceteris paribus, there's no technical advantage to using protobufs. The choice is almost always non-technical (business, platform, partnership, etc) or simple naivete. It's not harsh, it's just engineering.

> make apologies from authority (after all, Google can't do anything wrong, right?)That's not my position at all.  In my other comment (https://news.ycombinator.com/item?id=25586447) I explain how I've spent 10 years trying to improve on protobuf C++ precisely because I agree that some of these limitations are unnecessary.
> No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications.
I suppose it depends what you are comparing it to.  Almost every JSON library has the same limitations you mentioned, and yet many people find JSON useful for network applications.  But I agree that giving users full control over allocations makes a library useful in many more situations.
I think arena allocation is a pretty reasonable solution to the problem.  You can use whatever memory you want for the arena (stack, heap, static buffer) and you can constrain it so that no heap allocations are allowed.
Unfortunately protobuf C++ can't live fully within this arena model while it uses std::string for accessors.  Hopefully this can be fixed at some point.

> I suppose it depends what you are comparing it to. Almost every JSON library has the same limitations you mentioned, and yet many people find JSON useful for network applications.JSON is a human-readable format which is hugely advantageous to develop and operate in many settings. Protobufs doesn't have that advantage. Yet we're paying all of the same costs to structure the data with both. That's enough to posit JSON as a net winner over protobufs.
> Unfortunately protobuf C++ can't live fully within this arena model while it uses std::string for accessors.
Forcing std::string as a container for core components of a networking API (specifically for an accessor) is an exemplary demonstration of a lack of seriousness in a library.

> Forcing std::string as a container for core components of a networking API (specifically for an accessor) is an exemplary demonstration of a lack of seriousness in a library.im not super experienced with c++, so please forgive the maybe obvious question, but why is that?
(im guessing it has to do with allowing control over where allocations come from?)

In general I agree although I think it's fair to separate the protobuf wire format from the proto3 schema format which had a canonical json representation.

You can debate the design decisions all you want (and I agree `std::string` was a poor choice here)... but it's kind of absurd to say that a technology underpinning products used by billions of people every day is not "serious".

Why is that absurd? Products used by billions of people may just as well not have serious requirements. I beg not to debate semantics, indeed my own fault, but in the world of network software requirements can start to get very serious, straight through to userspace protocol stacks on DMA'ed device buffers.Would you go through all of the effort to rig an application with userspace networking only to return its results inside an `std::string` -- one of the few std containers where one can't even control the allocator if they wanted to? That's absurd, if anything.

> but in the world of network software requirements can start to get very serious, straight through to userspace protocol stacks on DMA'ed device buffers.Google does all of that when it makes sense, and yet uses it to push protobufs.  That’s the entire point: you are calling it absurd and non-serious, but this only shifts my opinion on you, not on Google.

If you are arguing that there exist use cases for which Protobuf is ill-suited, then fine, I obviously agree.But you seem to be arguing that Protobuf is bad because it is not well suited to certain use cases, dismissing all other use cases as "non-serious". That is offensive.

I asked you not to argue semantics, but you're doing it anyway. I didn't respond here to argue protobufs actually is bad, since all software is suited to something for somebody somewhere.In fact, I can just do that now: Protobufs is bad. You seem to agree, because you haven't even advocated for it -- just flatbuffers and Cap'n Proto and the phonebook of alternatives. Listen, I don't care who you've worked for or what you've done. As a scientist and engineer I'm totally ready and able to entertain critiques of my prior works without being offended. I'm offended that this is the quality of your participation here.

This is a really unprofessional and unreasonable approach (and I can see you're being downvoted for it).  Since you say you're a scientist and engineer, can you share your prior work so we can evaluate it?

Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers.In C, dealing with memory allocation is such a pain that C programmers still tend to avoid it. But C++, especially post-C++11, makes dynamic memory allocation much, much easier to deal with. (And GC'd languages, obviously, are easier still.) There is still a performance cost, obviously, but that cost almost never matters in application programming use cases, and is even negligible in many systems programming cases.
I do think Protobuf does too much allocation, but saying that libraries should not allocate memory at all is an outdated view.

Constrained embedded systems cover a broad range of things up to and including your phone. There's few things I hate to see more than a flat profile from memory allocation or cache misses.The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse.
[1] https://google.github.io/flatbuffers/

> Constrained embedded systems cover a broad range of things up to and including your phone.No, modern phones are certainly not constrained in the way I meant, and I don't think you could call them "embedded" either. The common programming languages used on phones are very memory-allocation-friendly.
> There's few things I hate to see more than a flat profile from memory allocation or cache misses.
I think you may be arguing a different point, or a different level of extremity of the point. Reducing memory allocation to optimize performance is a fine thing that everyone does. The person I was replying to, though, seemed to be asserting that libraries should completely avoid allocating memory for themselves.
> The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse.
Yeah... I'm the author of Cap'n Proto, which has the same property, and predates Flatbuffers.

> The common programming languages used on phones are very memory-allocation-friendly.I would hardly classify Dalvik or ART as "allocation friendly", they don't perform escape analysis and if you do it constantly you'll be in a world of constant hard GC pauses. Multiple times over the years I've had to build free-lists in Java to avoid this specific problem.
Same for C++ if you use one of the built-in generic memory allocators. The fastest new/delete are the ones that you don't call.
For what it's worth I tend to agree with the grand-parent thread. The lack of awareness of allocations, cache-invalidation via indirection are a significant contributor to why we see software clawing back hardware wins across the years on these platforms.

> I would hardly classify Dalvik or ART as "allocation friendly", they don't perform escape analysis and if you do it constantly you'll be in a world of constant hard GC pauses. Multiple times over the years I've had to build free-lists in Java to avoid this specific problem.I don't know what you're doing, but I've generally found free-lists to be a net performance negative in Java libraries. Time and again, I've been called in to "optimize" Java code that uses them, and usually by simply removing them I can get rid of the performance problems entirely.

Dalvik and ART have different behaviors than traditional Java VMs. Dalvik is aimed at low memory devices, ART trades some memory for speed but still is significantly different(i.e. it doesn't do escape analysis). As always benchmarking on hardware is the way to confirm this but I've had numerous cases where free lists or pre-allocated arrays has gained 10-30x performance improvements.Flatbuffers in particular excels here since once you have the ByteBuffer in memory you can immediately start accessing data without needing to do any extensive parsing.
Even the official Android docs are very explicit[1] on the allocation point. Allocations are not cheap and even with generational collection you still will blow the 16.6ms frame window for 60 FPS if any of your operations allocate excessively.
[1] https://developer.android.com/training/custom-views/optimizi...

Sure, with flatbuffers, I grok the case (though where possible I just use an off-heap mmap buffer for them anyway), but that's a very specific case.

You may be right - presumably iPhones have fewer GC pauses (though you can still have VM, loading, compression/decompression, network, and other pauses.)Be that as it may, lots of people still manage to use Android devices.

> Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers.Allow me to unpack this, because the object interface that all of these serialization schemes present is what increases complexity. It makes sense that the simple implementation would tend toward dynamic memory and not further compound such complexity on their interface. This is a flawed architectural premise taken by these library implementations.
On the receiving side, there is little reason to ever deviate from zero-copy/zero-allocate. These libraries are far too pro-active in deserializing incoming data into native object structures. It's not necessary to present the user with this; they should pique through the results lazily -- query into the data at their own discretion. This is important because it bounds the minimum requisite cost of receiving messages at ... zero. It costs nothing to discard trash. Whereas with these proactive serialization layers, every single incoming attack becomes an exercise against the deserializer. In a zero-copy/zero-allocate -- let's even say, zero-parse mere lexing of the data, software has more flexibility. In practice, that means passing pointers around to parts of messages straight off the wire to application functionality further up the stack.
On the sending side, there is little reason to ever force whole native object representations to conduct a serialization. In other words, don't force the user to build an std::map first and serialize it later. All input is streaming input. Properties do not have to be pre-buffered, they can be streamed. One can model any network serialization this way, sans perhaps canonical representations (sorted JSON keys, etc), and far more efficiently than with requiring arbitrary native structures.
This is not really a rebuttal about the merits for or against dynamic memory in and of itself. I know the research, thread-aware allocators can be pretty good -- even darn good, and the state of the art in GC is nothing to shake a stick at. The problem is that with better design it's just not necessary, and in the end it does have a cost that one should want to eliminate if possible. I'm certain it's quite usually possible, at least more than I see in a list like https://en.wikipedia.org/wiki/Comparison_of_data-serializati... etc

What you describe sounds exactly like how Cap'n Proto works. Obviously I like that design since I wrote it.But it does have drawbacks. The encoded size is necessarily a bit larger to allow random access traversal of the raw bytes (though it compresses well). The API to manipulate structures in-place is a little awkward, particularly on the writing side. And, you can't really use the generated types as mutable in-memory state, as people commonly like to do with Protobuf types -- as a result, a common feature request for Cap'n Proto is to support generating "native" structs with the ability to convert between those and the zero-copy types as desired.
Everything is full of trade-offs. I don't disagree with your design preferences but I do object to the extreme line you are taking on them. You said: "Protobufs isn't useful for serious network applications." That is plainly contradicted by the existence of a trillion-dollar company built on said technology.

> These libraries are far too pro-active in deserializing incoming data into native object structures.Sometimes a native object structure is what you want.

Once upon a time, we were concerned with handling network traffic at wire speed. I can remember when the AIX folks were happy they could implement the TCP fast path entirely in the interrupt handler.

Certainly you wouldn't use protobuf to encode TCP packets. It's meant for encoding application-layer messages, typically ones that aren't especially large but contain a lot of complex structure.

Where efficiency doesn't matter because they don't need to be processed quickly and cpu cycles and memory are free.

"that any library -- any library -- that forces internal dynamic memory upon its user smells bad"Cough.
While with embedded systems there definitely is a big thing about dynamic memory allocation, much as I don't like it, it's not like very popular and successful libraries don't do this. It's a pretty common and accepted practice, and there are standard idioms for how it is done.

The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use.  See more in our Twitter conversation here: https://twitter.com/richardstartin/status/134406813297822105...

"possibly multi-MB messages"Megabytes?
By "telemetry" does he mean "getting info on what our pumps out in the oil field are doing" or "snooping on users of a program"?

think of the type of monitoring/observability data that Datadog's agent collects

People tend to jump from telemetry straight to snooping on users even though it's just one possible case.A possible megabyte of data: A rolling in-memory debug-level log + stack traces with context + timing information, that you snapshot when the app catches and exception. Or a buffered set of timing stats from a minute of runtime.
The author mentioned implementing ddsketch for probabilistic quantile distribution. That's something you run on your own app/infrastructure rather than for snooping on users.

This over-assuming-of-shared-context is one of my pet peeves on this site. Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow.Of course, I'm also annoyed that some computer science PhD apparently googled around for a cooler name for multi-dimensional arrays and ruined the term "Tensor" forevermore.

> Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow.Did you even read the post? It's obvious OP works at Datadog ( they mention it and mention the work is for their employer), so telemetry as in OpenTelemetry ( the new standard around metrics, traces and logs).

Well, I started to read the article, thinking that it might refer to sending, you know, telemetry. The hackernews link was titled like this:>Don't Use Protobuf for Telemetry
so I thought I had a shot of reading something relevant to me. So I opened up the article:
>Protobuf needs no introduction, but this post argues that you shouldn’t use it for telemetry. The basic premise of this post is that a good telemetry library needs to be lightweight to avoid perturbing the application; inefficient diagnostic tools are self-defeating. Unlike other formats, nested Protobuf messages cannot be written contiguously into a stream without significant buffering. The post doesn’t argue to never use Protobuf, but that the trade-off made by the wire-format itself, as opposed to any existing implementation, is unlikely to work for lightweight message senders.
My field is related to robotics, where we actually send, you know, telemetry. Like, over a radio. What I'm complaining about is that this is arguably the "main" usage of this word, but people in the web space assume that like protobuf, the concept of telemetry needs no introduction: it means sending back statistics, metrics and logs about your web services. When he says "a good telemetry library" needs to be lightweight, there's zero preamble here to get me oriented.
You saying "did you even read the post" is exactly missing the point. Yes, I read through the post, and through context clues, discerned that this post isn't relevant to my interests. I just notice that people writing about cloud service technology never feel compelled to warn you that's what they're going to be talking about. They make grand proclamations, and you have to play detective to find out whether their claims apply to you. "Microservices are the future!" "Okay, does he mean like... for all of software? Or for SaaS companies?"
Imagine if the world was a different place, and the average programmer worked at a robotics company instead of a web tech company. And then you saw an article titled "gRPC is dead - DDS is the future". And then it jumped in and started talking about all these throughput metrics, and it took you 6 paragraphs to figure out that the reason they're done with gRPC is that they're shipping around 4K images at 60fps to dozens of microservices that all run on the same computer. And you'd go oh, well that's... irrelevant. That's how it feels, all the time.

To be fair, if you don’t want to read grand proclamations about opinions which make zero sense except for people in roles tightly related to surveillance capitalism, perhaps this isn’t the site for you. (Us...)

The practice of naming software or start-ups after real, useful terms from other industries is super annoying to me.And it's especially unconscionable for web devs, who should be more wary than most about global namespace collision (looking at you, browser window). At least have the decency to change the word slightly.

> What I'm complaining about is that this is arguably the "main" usage of this wordTelemetry is getting remote metrics. You can have telemetry in robotics, you can have telemetry for app performance, you can have telemetry for environment sensors, etc. They're all valid - I'm not sure why you say your field's usage is the "main" one.

Yes. Just recently I discovered that some web types refer to generating HTML from some other representation as "rendering". To graphics people, and artists, rendering is taking a description of something and turning it into a directly viewable picture. HTML is a long, long way from displayed pixels.Come on. There's a big industry out there working to push game and entertainment content through the GPU and onto the screen. That's rendering.

Rendering is what happens when fat is cooked slowly over low heat and becomes a liquid, rather than crisping up. What pixels have to do with that, I have no idea.

Software is now literally everywhere, doing everything.Picture if there was a website dedicated to Math News.  Some articles talk about how amazing Tangents are.  Some talk about Bayesian inference.  It would be a mess.
Yes, Hacker News is a mess.  That's pretty unavoidable.  All conversations take place within a domain - a culture, a set of shared assumptions.  And if you start talking with someone new, yes, it will take you quite a while to find out what assumptions they have that are different from yours.
Like, I wander in to an economics class, and they put the independent axis in the wrong direction.
I open up a computer graphics library, and it's using the left-hand rule.
One textbook says negative charge goes this way, and the textbook for a different course says it goes the other way.
Yes, people don't even understand that others MIGHT not understand their domain.
Yes, that is the world.

I read the whole post, and had to do a mid-post brain state reshuffle to get out of my assumed “pump in the field” context to his ”snooping on users” context.This for me, is because I’m playing with an ESP32 based project which uses protobufs, and I clocked into it from that mind set.
The intended use of “telemetry” was not obvious for me, until some way until reading the piece. Even the first mention of Datadog didn’t raise any suspicions to me, it was the use of Java that first had me wondering, then down around the 6th para the lightbulb went off when he said “a diagnostic agent which tells you what your application is doing” that I stopped and re oriented my brain.

If you are expecting multi-MB messages protobuf isn't the best option as the author highlighted. Often you can combat this by breaking a large message into a stream of smaller messages but for larger amounts of data you will want something else.

So, essentially, they're shoving a square peg into a round hole, and are surprised there's a bit of an air gap.

Here's a minimal Python protobuf parser that can extract basic information in about 50 LOC.https://github.com/kubernetes/test-infra/blob/master/guberna...

pb is also used in low-resource/embedded systems (to send the data to the servers). the processors typically have little to no cache and very limited RAM,Performing multiple encoding passes is the only option and each pass is as expensive as the first.

In these environments you'd typically take special care to keep messages flat, I think - in which case there's no problem with multiple passes.

Its a shame you can't share the code, we could really do with a modern C++ implementation. Having said that, protozero looks very interesting [1], need to find some time to look into it.[1] https://github.com/mapbox/protozero

You don't work for Blizzard do you? When I worked there engineers (not my team thankfully) designed their own alternative to protobuf to try and save more bytes over the wire, which in my opinion was a really poor decision. Rather than get on with actually adding value, they ended up pushing back multiple other teams deadlines while adding almost no value. It was a classic "not built here" mentality and doing engineering work for the sake of the work, rather than actually ask "what problem are we trying to solve here?"If protobuf works for Google then it essentially works for 99.999% of every other company on the globe.

If protobuf works for Google then it essentially works for 99.999% of every other company on the globe.I can't agree with this mindset.
Another commenter here pointed out the Google implementation is 20x slower than others, and 1.6MB for this kind of task feels bloaty.  Just because it meets Google's needs doesn't mean it's universally adequate.

Why exactly does a megabyte matter, unless you're trying to run on a 90's watch or embedded?

Maybe because "20X slower"

These are simply tradeoffs. For example, one can make very fast protobuf parsers/serializers if you prioritize that over the ergonomics of the generated code.

Well, transmitting a megabyte also takes a full second at 8 Mbps.

Nobody's transmitting this, it's the size of the library doing the proto serialization/deser

I did a search on this page and I didn't find the comment you are referencing. It seems very unlikely to me that google's implementation is 20x slower.

See tijsvd's comment about 1 microsecond vs. 50 nanoseconds.  (Perhaps I should have caveated that with "in some scenarios")

Which implementation of protobuf is 20x faster than Google's?

I think the general point is 'if it works for Google, you should feel relatively safe adopting it for your own use'. It doesn't mean it'll fit every single usecase.If what you need is something that is super efficient over the wire - or any other such requirement not filled by Protobuf - then maybe look for a different protocol altogether. Or design your own!
But for the rest of us - the '99.999%' - the trade-offs are well understood and we'd rather go for the tool we know than reinvent the wheel or use a less maintained tool.

No I don't work for Blizzard. When I did this, we were using a very fast custom wire format, but entirely hand-coded. Say flatbuffers without the code generation part. And having massive trouble with schema evolution.Another group had already evaluated pb and found it way too slow. They had designed something similar but faster. I wrote my implementation of pb to prevent this, and showed pb could be fast enough. It was definitely the right choice at the time, as it gave us C++ speed close to the old format, plus easy interop with other languages.
> If protobuf works for Google then it essentially works for 99.999% of every other company on the globe.
Uh.. no. Google is a massive company, but if you browse the comments in this thread, you'll find multiple remarks like "this was built for Google's servers". Google have specific use cases, and they build software for that. The software may well be lacking for other use cases. I can totally imagine Blizzard wanting to write their own implementation, think of the benefits of reducing parse time in a multiplayer server.

I remember watching a GDC talk of JAM (https://www.gdcvault.com/play/1018184/Network-Serialization-...), which was Blizzard's solution to network serialization for WoW back in 2004. The talk even had performance comparisons against protobuf and it seemed like a decent alternative that worked well for them. Was this new alternative an extension of JAM, or a full rewrite? I know they have millions of players monthly, but I can't imagine saving a bit on bandwidth could have warranted a full rewrite, especially when the majority of their traffic is from their own datacenters.

Yeah, the author seems to both understand the nesting issue, and yet not understand how it's not an issue.As per the format guidelines, you shouldn't have deeply nested structures... or at least if you do it should be in cases where only one or two layers of nesting are decoded at any given stage (I've done this with "envelope" messages).
Of course, there are ways to write serialized lengths of nested elements recursively, you just have to do it yourself. It's just not what you normally want to do, so the standard library and compiler don't emit that logic. If you are going down that road, you gotta ask yourself why.

If you truly wanted speed, why not Cap'n'Proto?

It didn't exist yet. We were evolving from simple raw messages with all sorts of problems.The alternative was something custom again, with better support for schema evolution, but pb was convenient due to existing implementations in Python (system tests) and C# (UI).

Reply on news.ycombinator.com