Offset: Goroutines ported to Python | Hacker News Reader

Offset: Goroutines ported to Python

docs.google.com

28 pointsscardine12 comments

Mildly related - it's been my opinion that Go's M:N threading system should be considered a workaround for a missing OS feature: the ability to manage and schedule native threads in userspace, without incurring a context switch. There are many reasons why goroutines would be better implemented in cooperation with the kernel - better debugger support, thread local storage, probably less scheduler overhead - but an important one is that it would make them easier to port to other systems than the Go runtime, such as this one. This is not just because they wouldn't have to reimplement the scheduler (which could be addressed with a regular library), but because they wouldn't have to contort every interface that touches syscalls into taking the userland scheduler into account. No need to put syscalls in their own threads or pass the context object around or anything, the kernel would just return to another goroutine while working on a syscall, just like any other thread.

I can dream.

Not really a work-around, even with great native thread support and tooling as you dream of (and don't get me wrong, I am all for it), ultra-light "goroutines" or "erlang processes" fit a different niche, you are never going to spin up 3+ million OS threads, you absolutely can (and do) in the ultra-light process space.

Having 3 million threads doesn't change the usefulness of being able to call into native syscalls and native libraries that use TLS, locks, etc. It does change the usefulness of native debugger support, since existing debuggers would have a rough time with 3 million threads, but it's not like goroutines currently have any solution to that - Go just uses a GDB script to implement goroutine commands similar to the thread ones. So I think it should absolutely be possible to spin up 3 million OS threads. The kernel needs to change to trust memory mapped into userland to keep track of threads rather than tracking all the data structures itself, and the toolchain needs to support split stacks. This would require a lot of change, but the result would be a lot cleaner than having two strongly overlapping concepts of threads.

Stackless Python support Go style concurrency with tasklets( goroutines in go) and channels for scheduling and data passing between them.

Pypy now has stackless support, which makes it the obvious candidate for doing CSP style concurrency with python.

It will not however, scale that concurrency to execute in parallel in multiple cores, since it is still bound by the GIL.

If you want to stay with CPython, you could look at Gevent.

gevent is based on an eventloop to do the scheduling, it's quite different than the scheduling done in offset and offset will soon support the possibility to launch your goroutin on different OS processes for //. More over gevent monkey-patch the standard library which make sometimes the integration with your code difficult, offset however use its own syscall module.

Also the implementation of channels or stackless in pypy is a little different than the one you can find in offset (or go for that matter). offset channels support buffering, and you can also select them. offset like the implementation of stackless in pypy is using the continulets to do the job when you are using in pypy and python-fibers on cpython which bind the stacklet lib from pypy as well.

Is this specifically for CPython? I thought Stackless Python already provided something similar barring Go-specific things like channels.

It is also working for Pypy. this is a different impementation from Python Stackless and also give you the possibility to select operations just like in Go.

Anybody have this content in a form that doesn't require a logged in Google account?

it's available on speaker deck too:

https://speakerdeck.com/benoitc/experimentation-in-porting-t...

Select 'File->Download'

You can't see any menus, any content, anything but a login page.

odd when i'm not logged to speaker deck I'm able to see the slides and download the PDF on the left :/

Reply on news.ycombinator.com