Hacker News Reader

Comment by sebastos | Hacker News Reader

Damned if you're not the most patient self-advocate for an open source project I've ever seen. I feel like I can always count on finding The Kenton Varda in the comments whenever a protobuf discussion comes up, graciously discussing design trade-offs and dispelling misconceptions.

Did you ever consider looking for official / commercial backing for Cap N Proto? It seems demonstrably better than all the alternatives, but I get the sense that it hasn't taken over because "nobody ever got fired for buying IBM". I know for certain I'd have an easier time pushing capnp in my own organization if it was getting that type of backing.

View on HN

Hah, thanks.

> Did you ever consider looking for official / commercial backing for Cap N Proto?

Well, I don't think Cap'n Proto is marketable (for any kind of profit) on its own. It's one of those things that has to be free and open source to get any adoption. So a corporate backer would need to get some indirect benefit from its development and adoption. Also, at this point my goal in building and maintaining Cap'n Proto is primarily to benefit my other projects that use it -- if other people benefit too, great, but wide adoption of Cap'n Proto is not an intrinsic goal of mine.

Right now, Cap'n Proto development is de facto backed by Cloudflare, as we are using it heavily in Cloudflare Workers and that is driving all the recent work on the C++ implementation. Currently, this use is internal-only and so we have little reason to build out support for multiple languages. But, as the Workers platform grows the ability to support increasingly complex apps and distributed systems (especially with Durable Objects), limiting applications to communicating with HTTP only is getting awkward. One idea that has been tossed around is to expose Cap'n Proto directly to apps. It makes a lot of sense: it would be very easy for us to support since we already use it under the hood, and zero-copy communications make a ton of sense especially for worker-to-worker comms happening within a single process. If we decided to do this, Cloudflare would become the commercial backer. But, there's also a strong argument to support gRPC directly in Workers, given the existing ecosystem. Would we want to support both? Maybe, maybe not. There's a lot of trade-offs still to think about here, and my goal would be to make the best possible product decision for Cloudflare Workers -- not necessarily for Cap'n Proto.

Thanks for the insight, that makes sense!

Perhaps this is too unhip, but I wonder - have you ever dealt with the OMG's Data Distribution Service (DDS)? I used to think it was too fuddy-duddy to even glance at, but the recent adoption of DDS by ROS2 made me take another look. DDS seems to have a lot of interesting properties, particularly the QoS and discovery mechanisms. What made me think of it is that DDS is really a free and open standard, but there are a number of commercial entities making money off of it. They provide DDS implementations for free, and then they make money by offering professional consulting and support. Maybe a Cloudflare-backed capnp could end up looking the same way. Anyway, I'm sure you have your head around the possibilities and tradeoffs, not sure why I'm rambling.

Eh, I'm not a big fan of the "provide paid support for open tech" model... it doesn't scale well, and it seems like it creates a perverse incentive to make the tech hard to use, to generate contracts.

Yeah, I see your point, and I have to admit that it's been kind of a turnoff as I've been considering DDS for my own company. The other thing it does is create incentives to create optional paid add-ons. Then you find the support company talking out of both sides of their mouth, because they're trying to sell you on how great it is that the underlying standard is free and open, but simultaneously trying to sell you on how their paid, proprietary add-on is absolutely crucial. It creates an unease, where you're never sure where the border between paid and free will lie.

Cap'n'proto has a number of drawbacks both from the 'one man band' pov. (Witness the two year lull when Kenton went sandstorming), an awkward API and limited language support.

Personally I don't think there is a perfect protocol because different people want different things whether self describing, easy/optimised memory management, zero copy, partial decode. The list goes on....

At a pinch, flatbuffers with flexbuffer evolution would be close to my goals but I'd much prefer having a meta description of messages and perhaps access ,authentication, transport security and use that (e.g. an OpenAPI v3.1 spec) to generate an implementation whether in protobuf, msgpack, JSON, ASN1 etc. whichever is suitable for a use case and using an appropriate transport whether quic, TCP, UDP.

Some of the high performance work I've seen uses ASN.1 on a very large virtual server at 100Gb line rates because the messages lend themselves to parallel decode.

I think Mike Acton had it right by suggesting things are tailored to the data needs and not overgeneralised.

> Witness the two year lull when Kenton went sandstorming

There wasn't really a lull; development on Cap'n Proto continued that whole time by the Sandstorm team in service of Sandstorm. There was an absence of official releases since Sandstorm always used the latest Cap'n Proto code from git. The same story continues today with Sandstorm replaced by Cloudflare Workers. TBH I should probably give up on "official releases" and just advise everyone to use git...

Wasn't a criticism, just an observation from a 'user' when a key bit of infrastructure is directed by a single or small team.

You are opening up a technology that you are crafting for your own purposes in the best spirit of sharing.

Personally I'm wary of relying on any project driven in this way.

Inspiring, but not something I'm personally keen on using directly.

I do feel grateful that you've given this as an option and I do feel slightly parasitic in not providing a tangible positive contribution instead.

Reply on news.ycombinator.com