I'm learning Python. Do someone knows a resource to learn the well-known tricks for "fast" Python?
A book, a website, a cheat sheet or even a MOOC or part of a MOOC?
A book, a website, a cheat sheet or even a MOOC or part of a MOOC?
This is even more relevant if you are still learning the language. Focus on learning it, and leave arcane always changing implementation details for after you know it well.
Also for all languages where you want to do performance work: Learn to use a profiler, so you can find the things that matter. And then selectively look what you can improve there. Even in Python, some hacks or slightly unergonomic patterns in a really hot loop can be worth a lot.
[1] https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpyth...
- Get the overall system structure in Python. Get the architectural design right, and the big-O stuff right.
- If there are bottlenecks, re-code those directly in C, cython, or similar. Or better yet, find libraries.
Python is great for expressing high-level operations and system design. It is also very easy to integrate with native code. I've never had much happiness in optimizing Python itself beyond that. Broadly speaking, code falls into three categories:
- Most code: Instant. Performance doesn't matter. Python
- Some code: Big-O(lifespan of the universe). Not worth building.
- Narrow slice of stuff in between: Don't do in Python, but use Python as the glue.
Anyway, here it is, this is the second edition of the book https://www.amazon.com/dp/1492056359
I read the first one at work a few years ago and was fascinated with all the goodies it shared with us.
In my humble opinion, it's a must read by all Python developers, experienced or not; everyone will learn something new just by reading it.
A couple of advises:
- the right algo will go a long, long way.
- know your data structures. E.G: assigning to a slice is ridiculously fast (even while unpacking), memory views may save a lot on byte heavy workloads, heapq and deque are underrated, etc. Also check out https://wiki.python.org/moin/TimeComplexity for big O notations on python builtin types common operations to understand what you pay for.
- know the stdlib. collections, itertools and functools all contain incredible gems.
- delegate. Python is a fantastic glue language, use it for what it's good at. Your database, your numpy arrays, your cache are all amazing are what they do, no matter the language. Let them do the heavy work.
- don't kill good perfs by ignorance. I regularly see people casting a generator, iterating on a dataframe, calling readlines() on a file or doing something else that is destroying the otherwise excellent perfs of their program.
- know the ecosystem. There are some very good fast libs out there: diskcache, sortedcontainer, scipy, uvloop...
- use threads to avoid blocking a GUI, multiprocess to share work between CPU and asyncio to speed up network operations. Each tool has a sweet spot. But threads are underrated, they work well for a couple of hundred parallel network operations, and most C libs will actually release the GIL, so they can use several CPU more often than you'd thin. Also, use pools if you can, shared_memory in 3.8 or mmap.
- sometime the dirty solution is just faster, like subprocessing to ffmpeg.
- The more recent, the slower. Sure, I love statistics, pathlib and dataclasses. In a regular code they are great. On a bottleneck however, they are very slow.
- the array module is not supposed to speed up the code, only save memory. But sometimes it does.
- printing to the terminal is limiting. Sometime your program is doing fine, the display is preventing it to go faster. At least check the flush.
- comprehensions are faster than alternatives.
- pre-allocating lists and dicts can help.
- measure. The austin profiler is your friend.
- rewriting the hot path in a faster language is likely more interesting than writing the whole program in it. A bit of nim or rust is easy to call from python.
- some python distribution are faster than others. I don't mean just pypy, but also regular cpyhon that have been built specially for some architectures. E.G: https://www.intel.com/content/www/us/en/developer/tools/onea...
- nuitka code compilation can speed up the code by a factor of 4, it's nice, especially for startup.
Do you have any numbers to back this up? Why would Pathlib be slow?
TL;DR: Path() initialization is very heavy compared to simple strings, and subsequent path operations produce new Path() objects every time.
The video then illustrates the point with a patch to the black formatter cache yielding a 40X speedup.