Hacker News Reader

Comment by palish | Hacker News Reader

However anything that requires some complex interprocess communication is essentially unworkable. You need to serialize and deserialize your data into wire-level protocols, which is often prohibitively expensive, even when you can get it to work.

Would you post more info about previous attempts at this? I was about to create precisely that. Literally today.

But if it's fundamentally unworkable...

View on HN

What's fundamentally unworkable -- serialization?? There are plenty of options in Python -- pickle, JSON (no graphs), or protocol buffers/thrift if you want something language agnostic.

Having a serialization layer in distributed applications is a pretty vanilla requirement... I am also confused about the "unworkable" comment.

The "multiprocessing" module (which I have heard some not-so-good things about) automatically serializes Python objects over byte streams using pickle or some such thing.

Serialization is very expansive compared to shared data structures. Even if not everybody is affected by that, it still makes serialization unsuitable as a general solution in this case.

I've been trying to tackle precisely that problem.

Would someone please help me out and point me at a way of efficiently exposing a read-only block of memory to Python?

EDIT: Looks like "MemoryView" introduced in 2.7 is the answer.

2.7 introduced memoryview, and there are a slew of C API routines to interact with it ...

as for shared memory, have you tried just mmap'ing the node from /dev/shm ?

Aha! That sounds like just the ticket. I hope it's easy to use. A million thank-you's, and a Salami.

EDIT: I just saw your edit. That's what I'm doing --- shm_open(), ftruncate(), then mmap(). A little annoying that it doesn't automatically kill the shared memory when the program terminates, but still...

It's pretty slick, and I've had good results using it in conjunction with a C program that generates data.

I'd advise specifically against using Protobuf -- Google's support for Python seems to be pretty bad.

MessagePack looks quite interesting (http://msgpack.org/) and seems to have quite a lot of binding (I don't the support level on each one though)

I've worked with protobufs in Python a bit, although not in production, and it seems fine. What trouble have you run into?

Well, the Protobuf PyPI package (which is two versions out of date) has not been able to be installed by Pip for over 3 years now. And yes, this is a known bug.

Reply on news.ycombinator.com