Python 3.11 Delivers
twitter.com
twitter.com
> This is a graph of CPU utilization for the web services that power PyPI. Today we upgraded from 3.10 to 3.11 and saw a significant and correlated drop in CPU usage -- nearly half.
>
> The CPython team has been putting a lot of effort into improving performance recently and it shows!
Being a Lisp developer (for 40 years), I am finally developing some love for Python after years of having to use it at work for deep learning.
For me the better performance/lower resource improvements in 3.11 (looking forward to 3.12) is a big issue because I compare to Common Lisp performance, which can be very good.
Off topic, but my new found love for Python comes from realizing that many programming tasks can be done so much faster in Python.
Same here. I came to Python because I'm lazy ;-)
I stayed because the language is elegant and the community is simply outstanding.
(time (loop for i from 1 below (expt 10 8) by 2 when
(zerop (rem i 3) count t))
versus python
import time
def gato():
start = time.time()
s = sum(1 if i % 3 == 0 else 0 for i in range(1,10\**8,2))
end = time.time()
print(s)
print(end-start)
gato()
s = sum(i % 3 for i in range(1, 10\*8, 2)) s = sum(i % 3 == 0 for i in range(1, 10\*8, 2))Sure the language isn't Lisp, but it comes close enough, only to be let down by lack of performance focus, and PyPy never got the spotlight it deserves.
So I kept using Python only for sysadmin stuff when targeting UNIX like workflows.
Finally, it is looking like I can eventually some day reach out for Python in tasks where performace matters.
There's a "freedom of expression" that python has that other languages such as Java/Go lacks. C++ can have similar expressive power by overloading operators, then this was of course eschewed by Java and Go. Go and Java provide a framework for writing code their way. Python doesn't insist you use OOP methodologies, if you don't want (e.g.)
I built a data conversion system in python using the ">>" operator to simplify translation of data.
temp_f = json_data["temp_c"] >> if_none(use=0) >> to_decimal() >> round(places=4) >> convert_to_f()
>>> i = 12
>>> type(i)
<class 'int'>
>>> dir(i)
['__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getnewargs__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__le__', '__lshift__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes']Wait till you get ahold of ruby. It is the most LISP like language I've found yet in terms of flexibility and what it can do for you.
Given the recent news on Nx and Livebook, Elixir may be a first-class language for ML very soon.
On a related note, Elixir feels a bit like a functional love child of ruby and python.
Here are a few examples: https://github.com/wojtekmach/mix_install_examples
Plus at a certain point the people you have to deal with only know python.
https://gist.github.com/ssrihari/0bf159afb781eef7cc552a1a0b1...
Edit: if by "learning Lisp" you mean lisps other than Clojure, I don't know :)
* "Python 3.11 is faster than 3.8" https://news.ycombinator.com/item?id=33345421 (378 points | 49 days ago | 294 comments)
* "Python 3.11 Performance Benchmarks Are Looking Fantastic" https://news.ycombinator.com/item?id=31640049 (234 points | 6 months ago | 85 comments)
The packaging story seems less, uh, put together. Is any consensus forming on that?
Edited to add: both Poetry and PDM do generate cross-platform locks.
That shouldn't happen. I wonder what could cause it.
I never looked them up too closely, but same versions of Python should result in same version of packages regardless of platform.
¹but note that Debian & Debian derivatives shove several standard library modules into individual packages, so you might need to apt-get install more than just python3.
Vanilla pip works well with the venv module. I still manually manage the majority of my virtual environments, with the virtualenvwrapper libary, although `source path/to/myvenv/bin/activate` also works.
Pipx is great for installing executable tools.
Non-PEP440 compliant libraries are also a pain in the ass.
Put both those facts together and you are playing with fire. In all likelihood you will eventually footgun yourself and end up with a python env superfund.
Just use virtual envs for everything.
I say this coming from over a decade of python experience and lots of time spent installing python in exotic environments.
Edit: Notably, not in other tools such as pip.
I don't treat Python packaging tools as my religion. If one doesn't fulfill its purpose, I use another solution. Whether that's a positive or negative for the tool isn't my concern.
Without going into all the available solutions for all possible cases, here's a list of use cases that Python packaging needs to cover:
- dependency management and packaging:
- abstract for libraries, concrete for applications [0]
- pure Python dependencies
- dependencies with binaries included
- packaging format: sdist, wheel, conda
- library repositories: public (PyPI), private, mirrors etc
- installing the Python executable
- multiple Python executables
- management of (multiple) virtual environments
for real work I use conda because it manages multiple versions of python well. Then I mainly use pip to install stuff, occasionally falling back to conda or conda-forge.
This has been the most reliable for me. I often have to trash my whole environment and start over after installing a few binary packaes like tensorflow.
I'd advise against this habit, even for "light" development work. I've been bitten too many times by some library installing into the wrong env or picking up the wrong path of a library.
Conda's generally a good move if you don't want to manually manage virtualenvs.
It is I/O bound but it's all localized. Those URL endpoints aren't making external API calls. For tests, Postgres is set to use Session.begin_nested with SQLAlchemy which takes advantage of Postgres' ability to use SQL SAVEPOINTs[0]. The gory details are above my paygrade (I found it while Googling), but the end result is it makes Postgres able to be really really fast when accessing your DB in tests. I don't think it really writes to disk but it makes your app think it did and you get the "true" outcome of running the SQL (ids get created, all of your SQL works as expected, etc.). It's not a mock.
[0]: https://docs.sqlalchemy.org/en/20/orm/session_transaction.ht...
Eventually Dalvik got replaced with ART, Android team started to care about JITing and AOT compilation, and all those native methods got deprecated as JIT/AOT code out of pure Java implementation started to be faster than the cost to jump through FFI infrastructure.
Back to the main thread, I'd say the acceleration in Python 3.11 would benefit Taichi users too - there are still parts of Taichi that runs in Python (such as constructing the AST of computation kernels), which will run faster with 3.11.
Couple that with the fact that python on its own without packages isn't that useful and you can see why this is going to be a challenge .
Multiple interpreters in the same process with their own locks is happening now, though
If the GIL is removed and variables are no longer volatile (i.e, changes are no longer made visible immediately to other threads) that seems like it would break a lot of code. On the other hand, keeping every variable volatile seems like it would be terrible for performance.
Maybe I'm missing some critical difference between Python and the JVM here?
The smaller problem is this does at a small amount of overhead even to single-threaded performance. The gilectomy project improved performance elsewhere so net performance is close to the same.
The bigger problem would be integrating this strategy with all the libraries that rely on existing GIL behavior.
So make it a runtime option.
It's getting tiresome that python performance is suffering because some can't be bothered to write thread-safe software. In 2022.
The closest anyone has come to removing the GIL is the Gilectomy project by Larry Hastings, and it's unlikely to ever be upstreamed unless it could be somehow made to work with libraries that rely on assumptions about GIL mechanics (eg numpy).
It was Sam Gross and he more or less achieved it:
https://mail.python.org/archives/list/python-dev@python.org/...
Or else? It's not like they're not trying their best - or don't spend the level of effort that they and their companies are willing to take...
I don't find multithreading in languages without particular support easy at all, but I have become better at it. It is possible and sometimes necessary. It seems like the prevailing attitude in the Python ecosystem is weird, a kind of sour grapes thing, i.e. "Python doesn't have good multithreading support, but multithreading is ugly and error-prone anyway and the alternatives are almost as good or better".
Guido discussing this recently on Lex Friedman's podcast. tl;dw, he is open to the idea, he thinks it will be painful, he isn't convinced the demand is high enough yet .
What exactly big gain-eating abstraction you have in mind that was added to Python at which point?
It's not like Python devs build abstraction towers beyond what the language already has an encourages (unlike, say, Java which has a well deserved reputation for this kind of towering abstractions, or say, web development).
If anything, a lot of Python effort is on Data Science and scientific python, where wrapped C/C++/fortran libs for maximum speed/minimum memory is the order of the day.
Note, that I’m not dumping on Python. I use it daily but I think it’s likely a poor choice for apps that primarily tax the CPU.
In order to be competitive with what?
Making Python faster is great because it improves every existing use case for Python, not because it now makes Python a good choice for situations where it wasn't previously a good choice.
The biggest one remaining is probably matrix-synapse.