Comment by tmsh | Hacker News Reader

Comment by tmsh | Hacker News Reader

Parent Full thread

Right, but the problem is that the root node is always going to be Python, and any code that uses Python is going to have to go through the root node (hence all Python code is really part of the root subtree). This becomes problematic if you have more than one queue that you're watching (e.g., input, network, graphics output, etc.).

Were it possible to have Python code be in some non-root node, it'd be great (e.g., for parsing text, or doing something algorithmic). As it is, when everything has to be fed through the interpreter, the design becomes top-heavy from a performance standpoint because the slow stuff is on top. And any time you want to optimize something but then use some Python, you're stuck (it becomes part of the root subtree again).

At least that was my experience. But Python does encourage interesting alternatives for performance (with generators, etc.). And if you're already mostly doing Python anyway (I'd say the threshold has to be 70%), then it seems to make sense (v. the opportunity cost of switching contexts between languages -- if you are spending more than 30% in C, I find it difficult).

But it's a really interesting issue. C, C++, and Obj-C all take a much longer time to develop up front. But if you refactor into libraries, maybe not in the long-run...

In my experience, having python as the root node for multithreaded C/python programs generally doesn't present a problem. When calling into external C functions through ctypes, the GIL is released such that multithreading at the C-level is unaffected. I personally use this feature heavily to perform numerically intensive work with multiple threads as directed through python with C doing the grinding.

Additionally, all IO code in the standard library and most C-extensions that I've seen release the GIL around blocking IO calls. This allows multi-threading to be used for the standard blocking IO use cases. Further, the computationally intensive routines in numpy and scipy that I’m familiar with also release the GIL to facilitate multithreading at the C-level

The GIL only becomes an issue if you want multiple threads to be executing Python bytecode concurrently. I don’t believe this is a common problem as one rarely runs computationally intensive code at the python level. Such computationally intensive work is generally performed through C extensions or through external C functions (as accessed through ctypes). As long as such C extensions properly release the GIL, multithreading at the C-level is unaffected.

These are really good points. I suppose the only problem I see is when the C modules develop their own hierarchy of types and then they have to interface with Python's hierarchy of types. That's really the only issue.

If what you're doing in C is numerical or computational in a functional way that doesn't need a hierarchy of types -- then keeping all your types in Python and calling into C for IO, computation, etc., makes a lot of sense.

Where I've run into problems is if the C modules themselves get sophisticated and start having scene graphs or hierarchies of shapes, etc. Because then it makes sense to mirror the hierarchy in Python -- but then you get real performance issues.

So I guess if you can ensure that your C modules have a clear functional interface (or at least one whose side effects are clearly defined) and doesn't involve anything but a shallow type hierarchy, then Python + C is all good for that. I.e., the C modules have to be very well-defined or have just about no 'code smell' (at least near their interface with Python). Arguably you can always do this if you make an effort to refactor your C modules. But yeah, because Python must always be at the root node, one is sort of constrained to serve the 'top', so to speak. And that may encourage a sort of top-down design which is less amenable to bottom-up programming (as mentioned in On Lisp), etc.

Reply on news.ycombinator.com