Python programming: We want to make the language twice as fast, says its creator
zdnet.com
zdnet.com
There's a huge amount of pure python code out there, powering websites, acting as the glue between C functions and providing business logic to pre-optimised frameworks. These are not likely to be targets for rewriting in another language, but that doesn't mean that their maintainers wouldn't like them to be faster and cost less to run.
Also, the Python's C extension story is much better than most other popular languages, so that's something to think about as well if you seriously need performance in your app.
Another approach would be to accept that Python is slow and that the use case is very different compared to C. Python could improve on the things where it already has an edge and which fits to its design choices.
It is no longer and open source project, some corporate developers claim all power and attention and actively intrigue against others.
This "project" will result in 25-50% speedup at most, probably slow down some C extensions despite promises to the contrary, and then be marketed as a great success by the faithful and the press.
It sounds as though Microsoft has some strategic goals centered around Python, if I'm not over-reading Guido.
[0]https://marketplace.visualstudio.com/items?itemName=ms-pytho...
If I had to choose between 2x faster Python and a modicum of static safety, I’d choose the latter. Nothing worse than running a slow process for half a day, only to find out you mistyped “pickle.dump” at the end. I know type hints are there, but somehow they never save me from such errors.
You don't need type hints for that, any old Python linter would have spotted this error...
Linters help, but there is so much they don’t catch in practice.
If you want speed, you need complete control, mainly over memory. Custom allocator, that kind of thing. It's painful and annoying, but I do that in cpp because that's what people do and it's a skill that you can find in the market.
So when do I use python? When I really don't care at all about speed. Glue code, and that's a lot of code, is best in python. Moving files around, pulling data from some Web server, orchestration. Easy to understand, terse, simple to change, portable. This is also why coding quizzes are best done in python.
Also things that have few types but need to be fast, they make sense as a python wrapper. Classic is numerical stuff. Set up some matrices, turn a handle that drives a native lib.
That makes a kind of barbell. Fast to write and simple to understand, vs slower to write and more complex. This is probably why you see so many ads in the trading world for someone who does cpp and python.
Making python itself faster won't bring it to where it can be used instead of cpp, and it already is just a wrapper where the real work is done by cpp. The glue code we didn't care about in the first place because it's never a bottleneck.
I wrote a simulation of sorts in python a few months ago, and a scenario runner to crank through as many simulations in parallel as possible. I didn't have any particular goal for runtime, but the faster the better. I spent a day replacing an iterative solver with a closed form solution, and that made it about 10x faster alone. I spent another 20 minutes turning classes into slots and got an easy 50% boost on top of that. I run that code for a couple minutes every few days on my 8 core laptop from 2020. On my 5 year old laptop with half the cores, the same task is more like 5 minutes. Doubling the speed of that code would be really nifty, in that it would make the 5 year old laptop take a couple minutes, and the modern laptop take less than a minute. I'd probably double the number of scenarios and get a more statistically significant result for what I'm simulating.
I also like to use Cura for 3D printer slicing. That's pretty much all written in python, and while it's gotten faster over time it's still pretty darn slow. If that doubled in speed, it would be an awesome upgrade.
Yeah, who would appreciate a doubling of speed, e.g. needing half the servers or serving non-cached responses faster, or their scripts finishing in half the time, and so on...
I mean, if there was tradeoff for the end-user sure, but as it is, I can't even understand the reasoning behind this comment.
>If you want speed, you need complete control, mainly over memory.
That I might be OK without the ultimate in speed (that I'd have with the above in C or Rust, etc.) does not mean I would not absolutely appreciate more speed in the things I use Python for now, especially if it came for "free" (e.g. not with Cython or having to wrap C extensions, etc.).
Would you be saying the same ("doesn't matter") if Python was instead made 2 times slower?
I mean I guess at some point (5x, 20x, 100x) it would be unbearable, but yes.
It's nice for the tool to get better "for free" but if it's so far from becoming the right tool for certain tasks, what's the benefit?
Also it might not be "for free", I don't know what Guido is planning, but if you look at when people try to make Java or c# fast, it changes how you use the language by a lot.
So yeah it's nice for python to get faster but I won't know the difference at 2x or 0.5x, because I just don't use it for things where that matters. YMMV.
How about 30 seconds of my time saved everytime I run a 1 minute current script? Which I might do dozens of times every day?
(Actually I run huge pipelines that include pure Python processing, and I would absolutely love a double-the-speed raw python).
Oh wait, achieving native speed is close to impossible for a dynamically-typed language.
Also, do you really think the Python team should just give up on performance just because faster languages exist?
Should Tesla stop building cars as soon as someone builds a faster car than them?
nimpy https://github.com/yglukhov/nimpy PyGotham talk: https://www.youtube.com/watch?v=IVgNVJdizHg another blog post: https://robert-mcdermott.gitlab.io/posts/speeding-up-python-...
Whereas Python has tons of libraries, millions of programmers, tons of support, tons of documentation and books, and billions of lines of existing codebases...
Python was like that once, too. Now we have the necessary experience to understand what are crucial aspects of the language that are missing but are extremely difficult to change post-factum, such as removing the GIL. Nim has no such constraints, and while building the ecosystem takes time, I have no doubts it will progress smoothly as Nim is such a pleasure to use. Even for such a young language I feel we're already in a good shape: https://github.com/nim-lang/Nim/wiki/Curated-Packages.
Nim wants to be its own thing, and posters are quick to point out that Nim isn't Python, never was intended to be, and never will be. That's what the DFL wants, which is fine. But IMO, creating a language that is closely-related to Python, arguably one of the world's most popular languages, and allows / encourages porting large Python codebases, seems like a pretty worth goal. Now that Nim has already made a lot of decisions to make that difficult, it can't pivot to that goal without breaking all the existing Nim code. Too bad: seems like a big opportunity lost to me.
well they do state "copying bad design is not good design" in the main page. Actually porting Python code to Nim is not that difficult especially when people care enough to create pages like: https://github.com/nim-lang/Nim/wiki/Nim-for-Python-Programm...
Python is something like 40 times slower than C, lisp gets much closer with something like 4 times slower (all of this heavily depends on what you are doing obviously, so just rough ballparks here).
I think it's best to accept Python is just slow, and to use another language.
It's actually quite possible, and depends on the scenario.
A dynamically-typed language can analyse actual runtime type use and specialize function calls with a JIT, for example. It's also possible (and there's work in this area) to keep the specializations around for quick startup in the next runs... (and of course, drop them if they no longer hold).
That's how Javascript is much closer to native than Python - in some cases just 2-3 times slower than C. Lisp did very well too (better than more than a few static languages).
And that's with the extreme meta-programming / reflective behavior (which both Python and Javascript have). With less such, one could make a dynamic typing language that can be trivially JITed to run very close to native.
It would be interesting to see what kinds of speeds a scripting language that is actually well suited for JIT would be able to achieve if it had the same tremendous amount of effort put into optimizing it that Javascript has received.
- Strongtalk from the 1990s
- Pharo Smalltalk (Cog VM)
Also worth looking at
- Chez Scheme
And
- JavaScript in every popular modern browser.
And
- Luajit
I recently learned enough Go to rewrite some small tools for work that weren't fast enough for our current load and got 5x to 10x the performance. Took me 1 week staying 2h after work to learn Go from scratch for this. Goroutines and channels seem easier to manage than asyncio. Same numbers of lines. My coworkers who don't know Go understood the code just fine.
The only thing I'm not sure so far is the github packages. One one hand I can deploy anywhere and don't care about dependencies, on the other with python I got automatic and vetted updates straight from the linux distribution.
This is the problem that above initiative wants to fix.
If you're aware of the issues and have other options, you wouldn't look to Python for performance unless it was a case where Python is just used to script pre-existing C libraries (ex: machine learning). For general performance, you would choose Go over Python. If you choose Python, it will be for some other reason such as ease of programming, extensive and well-developed libraries, ease of hiring Python programmers, etc., where Python is better than Go and fast enough.
I've always loved Python and have done a LOT of it (professionally and privately) over many years, but Go has replaced most of it. Having fast, concurrent apps statically compiled into a single executable is a huge improvement for many of my uses of Python, esp. server-side apps and local command line utilities. But I would still prefer Python as a teaching language, for machine learning, for quick one-off tasks (ex: format an ugly chunk of copied data into C++ source code), and possibly for code that I wanted others to write and maintain for me. (Python programmers are easier to hire than Go programmers.)
Many things can be "fixed" if you choose to go to another system. Good developer environments limit the supported tools, languages and packages. This comes from several issues:
* When something goes wrong at 2 a.m., is the developer available going to be able make a reasoned decision how to fix the program?
* Can you hire people with a mix of Python and Go experience?
* If you want performance, why not choose a faster variant of Python so you stay in the same skill set?
* You successfully built a Go program, but did you document your development? Did you set up a standard environment? What about CI/CD and automated testing?
When you leave your position (and you will someday), who is really able to take over managing that software?
> My coworkers who don't know Go understood the code just fine.
Understanding and groking are two different things. I'm sure they looked at the code, nodded their heads that it makes sense; however, I suspect that if you asked one to modify the program with a new feature that they suddenly will see all the things that they don't know, from getting their environments working to subties in the language's mechanics.
Environments are standardized to prevent having these types of issues between workers. Software development is ultimately about supporting the people and business goals in the company.
"Just use multiprocessing" just doesn't work in this situation. We can't afford to multiply our memory consumption by 128.
We've already invested many months of developer time just to allow some parts of the data structure to be allocated in shared memory. But this has been highly complex (e.g. no more std::string for us), and it only allowed us to parallelize about half of the Python scripts; and the gains on the overall execution time have been fairly limited.
I'm starting to think we will have to rewrite 50000+ lines of Python code in another language, just so that we can parallelize something that would already be embarrassingly parallel in any other language.
I'm hoping that the "subinterpreters with separate GILs" will work out at some point; but I'm doubtful this can work as long as Python doesn't break compatibility with existing C extensions.
And yes, on Linux `fork()` trivially solves the "share complex C++ data structure for Python multiprocessing" issue. Unfortunately most of our customers use Windows :(
Maybe you can run your application in WSL2 with `fork()` on Windows.