Pyrobuf: A Cython Alternative to Google's Python Protobuf Library
github.com
github.com
The thriftpy project implements a pure Python parser for thrift files that dynamically generates Python modules without a build/codegen step. Also has Py2, Py3, and PyPy support.
Awesome library for this sort of thing. Includes a Cython impl of Thrift parsing/serialization, too:
I know that Thrift and Protobuf were developed / publicly released around the same time (~2008). They both have serialization and RPC approaches. They both have an IDL format and a compiler for codegen in static languages like C++ and Java.
Thrift was adopted by the Apache O/SS community and so it can be found in some Apache projects, e.g. Thrift is used as an interop format in Apache Storm and as a serialization backend supported by Apache Parquet.
Thrift has had a lot of client libraries for different languages come out over the years, so it tends to work everywhere.
IIRC, CapNProto and FlatBuffers were each built as fresh takes on protobuf. So I imagine they are similar, with IDLs and serialization approaches of their own. I took a quick look at CapNProto's Python module documentation and it looks very similar to thriftpy in philosophy, but I don't know its intricate details as I haven't used it.
This guide to Thrift covers all the important details of its schema approach and is meant as a cross-language guide. It should be the official docs, but alas, it's not!
Second problem is Capnproto doesn't have structs, ie. you want something like struct Vec3 {x: float, y: float, z: float} in CapnProto you have to either make Vec3 a ref type (+ pointer size and lookup indirection for each instance) or inline it manually (especially annoying given that you must manually number CapnProto fields, maybe this is a benefit in some use cases but it's of no use to me and only creates work).
CapnProto is much bigger as well - it includes it's own RPC/distributed object protocol and library - the serialization part isn't separate they are in one library - that's taking in a lot of useless code that you would have to figure out if you decided to maintain it.
Also worth noting FlatBuffers on Python are actually not that fast because they are pure-python - there are Cython versions in the PR (at least last time I checked) - this doesn't matter in my use case but worth pointing out. And the Python API is very very unpythonic (C style API with even the naming completely off from PEP8) compared to CapnProto which is excellent !
I needed a cross language typed serialization format for binary files and FlatBuffers is better for that than CapnProto IMO.
Although Cap'n Proto's C++ implementation is hosted in a single git repository, it does compile to several distinct libraries. You can use just the core serialization/deserialization part, libcapnp, if that's all you need. There are separate components, libcapnpc and libcapnp-rpc, for dynamic reflection and the object-capability remote procedure call system.
It would be really nice if there was a pure Python implementation that didn't use tons of unnecessary metaprogramming.
Your comment wishes that there was a pure-Python implementation that doesn't do metaprogramming. That would imply doing code generation, but having the generated code be more "concrete", so-to-speak (containing all of the actual implementation).
However, a different comment (https://news.ycombinator.com/item?id=10762101) admires the thriftpy project which doesn't have a codegen step at all. This is in some sense the opposite of what you want: everything becomes metaprogramming.
I work on Protocol Buffers at Google, and I frequently observe these kinds of opposing feelings about how protobufs ought to work. One thing I've learned is how incredibly difficult it is to please everybody. Another good example of this is whether to be pure-Python: some people (like you) want that. But it's nearly impossible to make it very fast, and lots of users are sensitive to speed (you can find various comments and blog articles complaining about the speed of Python protobuf).
The thriftpy module dynamically generates a Python module that is based on parsing the thrift schema. That happens once: at module import time. After that, Python has cached the module in sys.modules and it doesn't need to be evaluated again during the program's runtime.
The module contains efficient Python classes that are Python bytecode just the same way "compiled" classes would be. But, thriftpy has Cython implementation of Thrift protocols and transports that compile down to C, not Python bytecode. Thus, the thriftpy impl's are not only more dynamic and less cumbersome than the codegen alternative, but they are also faster.
In the Python community, we prefer to take other approaches to achieve execution speed.
Dynamically generating a Python module is not "metaprogramming". It is not black magic. You can generate a module yourself simply by instantiating a Python module type. Hooking the import statement is an officially supported part of the language, and done by many libraries.
This may all seem very worrying to a C++ or Java programmer, but in the Python community we have been doing dynamism with execution speed for about 10 years now, and we haven't regretted any of the results!
It's interesting though that you describe this approach as more Pythonic, because one of the most outspoken critics of this approach is a hard-core Python guy that I work with who has lots of experience with the Python ecosystem. He feels very strongly that it is more Pythonic to have very flat/concrete generated code that is transparent to the reader, and really does not like the idea of hooking import and generating everything at runtime.
This is exactly what I am describing about how it is hard to please everybody. Different people appear to have fiercely different opinions about what is idiomatic.
When we wrote the Ruby protobuf implementation this year, we took an approach much more along the lines of what I personally prefer. The extension is mostly implemented in C, and it's very easy to build types at runtime. It doesn't directly import .proto files (which I would have preferred) because there was still some desire from others to have some kind of code generation. So the approach we took was to use a Ruby-like DSL for describing the protobuf schema. But really this is nothing but a translation of the .proto file into the Ruby DSL. ie. take the .proto file:
syntax = "proto3";
message Test {
int32 foo = 1;
double bar = 2;
Test test = 3;
}
require 'google/protobuf'
Google::Protobuf::DescriptorPool.generated_pool.build do
add_message "Test" do
optional :foo, :int32, 1
optional :bar, :double, 2
optional :test, :message, 3, "Test"
end
end
Test = Google::Protobuf::DescriptorPool.generated_pool.lookup("Test").msgclassNot wanting code generation might come from people who don't like the additional build step and artifact deployment. I don't like that either, but what I mean by "less metaprogramming" is that the code that's generated currently makes copious use of descriptors, metaclasses, and other "complex" things in a way that makes it extremely JIT-unfriendly on PyPy. We had to completely abandon it in favor of using protoc-c wrapped via CFFI. So I care about speed (almost entirely -- protobuf powers systems for us that do ~350K messages per second).
(I do though think it would be possible to satisfy both complaints simultaneously. I spent about 5 minutes trying to write a codegen-less, pure Python implementation: https://github.com/Julian/pb/tree/master/pb but I didn't find the Proto3 docs mature enough to figure out the binary protocol at the time. Not sure there's enough there to see what direction I was trying to go in, but I have given this a shot before).
And thanks for the reply, I do agree that there is some conflicting concerns involved here.
I am curious what led you to the conclusion that descriptors and metaclasses were making Python protobuf JIT-unfriendly. I ask because most of the "meta" stuff happens at startup, when you first import the module. It uses a metaclass to generate a bunch of Python methods, but after that they are just regular Python methods, and should be as easy for PyPy to optimize as anything else. There is a bit of reflection happening in some code-paths (like the __init__ method does loop over a list of fields to decide how to initialize them), but the metaclass at least is pretty much gone after import.
For this reason, I suspect that even if we ditched all the metaclass stuff, you'd see PyPy performance pretty close to what you're seeing now. The only thing I could see potentially making a more significant difference is if there were generated parsing code that switches on field number, instead of looking up fields by number in a dictionary. But given that Python doesn't have an actual switch statement, this might not be an improvement at all.
The python code that's generated is ok, but the problem is that everything is very dynamic. If you want this to work fast on PyPy, it should really generate a bunch of getters and setters that use attribute access and not go through generic __getattr__ that does some dictionary lookups. Additionally a lot of code is written in a way that creates a lot of temporary lists iterating over all fields and double checking stuff - this is bound to be way slower than a simple C/Java stuff that does the very simplest "check type - attribute access" sort of stuff.
protobuf, swift, etc happened when an advertising company thinks it is an engineering company
When I search the web for information about asn.1, I find very little that is of practical use. What library should I use? Why should I use it? How does it benchmark in comparison to the other tools? I've seen a few asn.1 library webpages and they all seem to take it as a given that I'm just looking for some way to deal with asn.1 data. They don't bother to try to convince me that their tool is efficient, or even that asn.1 is the right choice for my data in the first place.
if you are curious here is a good blog post https://ttsiodras.github.io/asn1.html
i am working with asn.1 the protocol MMS
And yes, "good marketing" in the form of readily available and well-documented libraries for the languages we use is a very important factor.
I bet the telecom sector has their battle-tested libraries for ASN.1, or at least for the parts they use. Are they open-source? Are they available for all languages wanted? No? Then why would I use ASN.1, just to use a "standard", if it means using worse code or writing it myself?
The data format isn't hard. It doesn't particularly impress me that asn.1 has worked for 20 years when I've seen hand-rolled formats do the same. The hard part of the process is defining the actual messages anyway.
The part that is valuable is the library and the tooling that makes efficient serialization easy. If it's not easy, it's not useful because I already know how to do it the hard way.
The marketing has a point, by the way. It's an indication that there are people who are willing to put in effort to get others to adopt their tooling. That's a good indication that when I have a problem, that there will be somebody else who's willing to put in effort to help me. Given that I have zero budget to pay for support, that's important.
Thanks for the link, though. There's definitely some useful information there.
Integers being encoded as signed is a nightmare for cryptography, which makes no use of them.
Yes, asn.1 is the all signing, all dancing serialization framework. Protobuf's aren't. That isn't by accident.