Hacker News Reader

Comment by ggm | Hacker News Reader

I get that if I adorn my python3 with types the right way, cython can uplift C betterer, but if I don't adorn my python3 and only use core imports (no pip) do I get any benefit?

I have some humongous (for me) dict lookups I am doing on strings, counting in ints, and if I could 2x or 3x faster without having to recode I'd love it. The Cython web is directed at people willing to recode, I am ultimately willing but I wish there was some low hanging fruit evidence: is this even plausibly going to get faster or is dict() walk cost just "it is what it is" ?

Oddly, it looks like across pickle dump/load I get some improvement. Does pickle restoring consume less memory than raw-made?

View on HN

You will ~likely get some improvement by directly putting your python code in a .pyx file and compiling it (without using any cdef functions). In most cases it will be as if you were calling the Python C API directly in a compiled C file. In my experience, the biggest difference will be a reduction in the function call overhead so it is better for tight loops.

I wouldn't expect a full 2x in every case. But it can also be very significant in cases where Cython deduces the type of local variables.

Cython 3 also uses python type annotaions to infer C types now too, so if you add python types you might see surprising cython speedups.

Coming back to this thread, I am running one now. It's too early to say if there's a speedup.

But I want to note it is ridiculously under-documented how to make what I would call a functional a.out from this toolset.

You basically are assumed to run python3 >>> import 'mything' and then run mything.main(args) instead of getting a functional commandline tool which "just runs" -the stepover is small, true, but I would have thought the PRIMARY use case here is commandline executable, not "embed .so in my interpreter"

There are a tonne of stack overflow "how do I" and all of them are really whacked: everyone trips over the same problems of :main undefined and a swag of other problems trying to call (g)cc themselves.

I guess it’s mismatched expectations. IMO the primary use-case is making python wrappers for C/C++ code or converting prototype Python code into a fast C extension for production use by adding types or progressively converting code into a language that is a mix of Python and C.

That's fair. Many happy customers so it's very likely I misunderstood the mission and vision.

My primary experience in Python3 is to minimise REPL and get to commands which do stdin -> <process it> -> stdout and gang them up, or uplift discrete commands into yield() pipes inside the same interpreter.

I guess I fell into the trap of going from the particular to the general.

It’s designed for building libraries really. You write functions in the foo.pyx file, Cython generates a foo.c file which gets compiled to foo.cpython-39m-x86_64-linux-gnu and then in your script you do “import foo”

I think in your case you might get a better benefit of using pypy and its jit.

The way I look at Cython is that it is a tool that allows me to write C code using python syntax. The biggest gains you get is when you use annotation to specify use simple (C) types with their drawbacks (like overflow etc) and or python features like exceptions, GIL etc

Thanks. I'm exploring that option, fixing bugs as I go too!

If you're happy to add some types you might also get some speed ups using mypyc. Another option might be to switch your dicts to pandas series; you might be iterating over several lookups in python and with pandas you might be able to switch those into functions written and compiled in C.

Did you try pypy.org ?

Yes, and I am still exploring. It is possible I have some scaling issues in dictionary which are not about compiled or not but about the cost of mutating data in a dict and gc costs.

Things like key lookup are already well optimized. So finding a more efficient algorithm might yield easier performance gains.

If you have for loops inside for loops, that's one place to attack. dict iterators, etc.

Reply on news.ycombinator.com