Hacker News Reader

Comment by amelius | Hacker News Reader

The problem is that if you don't port the GIL, you can't port the C API, and thus you can't support most of the Python library ecosystem.

View on HN

Are there really that many Python libraries implemented in C? Genuine question - I thought there were a few very high profile ones, but I always figured there weren't many (like 10% or less, total finger-in-the-air guess)

"there were a few very high profile ones"

The absolute number (or fraction) implemented in C is less important than the high profile ones. numpy/scipy alone are used in a huge number of projects so if you can't support those two libraries alone it's pretty much a non starter.

Oh I’m not downplaying them, if they were small in number and minor then I’m sure CPython would’ve happily dumped the GIL already.

The number of them isn't as relevant as their usage. Lots of data science related modules are effectively wrappers around C code. Here's a few:

  NumPy 
  pandas 
  Matplotlib 
  TensorFlow 
  PyTorch

These modules are critical to many of the workloads using Python (ML / AI / Data Science)

Right this is what I thought - a small number of popular ones, rather than an overall majority across pypi packages

In the data science space, which is probably one of the biggest users of Python, an interpreter that doesn't support these is dead in the water.

I think the question is whether it would suffice to port the few most popular native modules to directly use this project's bespoke Rust API instead of the standard CPython API. This'd probably be less work than building a real emulation layer for the CPython API, especially if they don't want to include the GIL. But if Python users need a lot of different native modules and not just a few popular ones, then that won't suffice.

> Are there really that many Python libraries implemented in C?

There are a lot that use the C API, even if some of those aren't implemented in C (some are actually implemented in Rust.)

This is what I meant, “implemented in C” and “use the C API” are basically the same thing at some level - if they depend on the C/Python interface at all

The ones that are fast are implemented in something other than Python. Most of them C, some Rust, but all native code. Python itself is just too slow to be useful in many scenarios.

Porting the GIL is trivial compared to what all it would take to support the C API

Reply on news.ycombinator.com