Hacker News Reader

Comment by ambivalence | Hacker News Reader

If you want real irony here, it's this: powering an app called Threads with Python that's got a global interpreter lock which makes threading hard.

Get it? Threads, threading...? OK, I'll see myself out.

View on HN

Now that is Meta!

Gotten burned by that. Have a python utility that calls something that's very CPU intensive (GDAL) and I literally have to wrap this python process in java for real concurrency. Not a python expert by any means but I'd love to know what other people do when this happens. I'm assuming it's something revolving around containers and messaging (blech)...

Is your problem that the c part of GDAL isn't concurrent and you are calling it multiple times, or are you blocking on something you shouldn't?

Python calling a concurrent c thing is super common. For more complex things there is multiprocessing, but that essentially is simplification of IPC.

Had a massive amount of images that I was trying to process for a specific purpose. I wound up spinning up multiple python processes within a java wrapper using concurrent linked queues. Got the job done but it's not the proudest moment of my life.

Why did you need the Java wrapper? You can orchestrate stuff like that within Python. The multiprocessing module.

You could have done this with the multiprocessing module. It interfaces like the threading library but it's launching new processes for each "thread". I've done a shitload of image processing in Python with it. You'll end up with a 100% Python version of what you did.

Multiprocessing does basically the same thing, but in python.

I thought Python calling a native lib was the one time you could multithread your way around the GIL if you were cpu limited? Are the GDAL bindings not releasing the lock or something?

Python works around this by having multithreading through multiprocessing and IPC. its made pretty easy to use with the futures builtin module but is finicky/slow/hard to support.

I guess PEP 703 has to be auto-accepted now. I don't make the rules.

Meta does have their own higher-performance JIT-compiled implementation of Python, but I don't know if it still uses a global interpreter lock.

... wait, what? Did this morph out of HHVM? Any links on this?

Click on the link, it's linked inside.

Seems to be behind login.

https://github.com/facebookincubator/cinder

Python does threading fine than most languges. Not every language supports native threading.

Reply on news.ycombinator.com