The Cost and Complexity of Cgo | Hacker News Reader

The Cost and Complexity of Cgo

cockroachlabs.com

96 pointsorangechairs6 comments

Interesting! I wonder if someone can help me with a problem I'm facing with cgo. I'm trying to include go runtime as apython module (.so) and the code appears to work fine as a standalone program. But when I try to run it from a pre-forked process manager (say Celery for python) the code gets stuck at runtime.futex (from strace). I guess some interplay of python and go runtime w.r.t goroutines. I feel Go is not meant to be embedded in Python runtime.

Code here: https://github.com/lazyfunctor/pymosaic

Edit: Added last line

"I feel Go is not meant to be embedded in Python runtime."

I suspect that it is going to ultimately be your core problem. Even if you make it work with specific versions or specific modules, it'll always be fragile.

Like many modern languages that embed an event loop into themselves one way or another, the Go runtime expects to fully own the target process. It makes it difficult to play well with others, because the "default" C-based runtime provides so few guarantees that it's difficult for two otherwise C-compatible runtimes to "play nice" with each other. It's part of why the JVM and .Net are so interesting for cross-language development; the VM provides a richer, better-specified "base layer" that multiple languages can use, enabling much easier interop.

It's a death-by-a-thousand-cuts sort of thing; hypothetically nothing really prevents this, but in reality you're probably not going to want to take the time to make it work which may very well involve custom modifications of either runtime, when you should probably just go ahead and set up some RPC connections. (JSON's a nice default to prototype quickly, but depending on your needs you may want something like protocol buffers or the half-a-dozen competing libraries.)

The "C managed memory" problem is definitely ugly. Some libraries and APIs allow you to protect yourself from this by giving you the option to supply the memory yourself, instead.

If your library can be used completely with you managing the memory, you can use Go's allocator. E.g. libopus:

    size := C.opus_encoder_get_size(C.int(channels))
    mem := make([]byte, size)
    p := (*C.OpusEncoder)(unsafe.Pointer(&enc.mem[0]))

This way, all memory is on the Go heap.

Not every C library will support this, but those who do make CGo a lot more bearable.

Thank you for sharing. I think you mean 1000 not 100 "100 goroutines come pretty much for free in Go;"

[cockroach labs employee here] Post corrected. Good catch. =)

Thanks for the pointers, cockroachlabs. Your posts really help developers building real applications on Golang.

Reply on news.ycombinator.com