gh-116167: Allow disabling the GIL
github.com
github.com
[0] Multithreaded Python without the GIL https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD...
[1] Github repo https://github.com/colesbury/nogil
There are so many different attempts at solving this problem that that the last time I wanted to try one of them, I found myself overwhelmed with options. I chose taichi which is pretty fun and easy to use, although somewhat limited in scope.
Taichi is really underrated, it works across all platforms (including Metal), has tons of examples and the code is easy to write. And lastly, it integrates with the ecosystem and doesn't displace it.
great demo reel of what Taichi can do, https://www.youtube.com/watch?v=oXRJoQGCYFg
https://www.researchgate.net/publication/337118128_Taichi_a_...
https://gigamonkeys.wordpress.com/2009/10/16/coders-c-plus-p...
Given that so many of the criticisms were about C++ being over-complicated, I do worry about languages just becoming more and more difficult over time as everyone wants their pet feature added, but due to backwards-compatibility concerns old/obsolete features are rarely removed. For example, take Java. I think that a ton of goodness has been added to Java over the decades, and for people who have been working with it throughout, it's great. But it feels like the learning curve for someone just getting involved with Java would be really steep, not just because there is just a ton of stuff, but because without having the context of the history and how things were added over time (usually with an eye towards backwards-compatibility) it feels like it would be hard to wrap your head around everything. If you're writing your own new program that's not really a problem as you can just stick to what you know, but if you're getting into an existing codebase that could use lots of different features it feels like it could be daunting.
It's been quite a while since I've programmed in Java, so I'm just speculating, but would be curious how other folks relatively new to the language in production environments find the learning curve.
As someone in the boat you mentioned (sort of) the short answer is modern Java development for 90% of tasks is not complicated at all: it's very much like any programming language used in a bizdev/corp environment -- you are mostly using a framework and a bunch of DSLs. Almost everyone uses Intellij and Gradle for IDE and build, and Junit5 or Spock for unit testing. I passed a technical interview mostly on Spring Framework concepts knowing almost nothing about it, nor having ever used it in production by simply just having the documentation open while I was being interviewed so I could look up the answers. Any language that is popular is going to have frameworks with decent documentation that help you be productive quickly, so I just jumped in doing Spring. The java stuff came as needed, or I referenced something like Effective Java (great book), or a Baeldung article. Java world has made some great strides since the 2000's and early 2010s of XML chaos. It took a while, but I feel like it's in a really good spot and getting better.
As an aside, if it hasn't been mentioned to you before, if you like simplicity in a language, but still incredibly productive, you might enjoy Go.
I've also been looking at Go quite a bit, especially from a lot of commentary even recently on YouTube, as well as plenty of time during work hours to experiment with languages I'm not familiar with. I've been deciding between Python and Go, and I picked up Python very quickly for the language, but ultimately I think it's the libraries and ecosystem around both that will decide it for me. Just good to see another vote for Go, especially when comparing to Java. I feel like I'm in the same boat as you seem to be, where it's all just syntax and learning frameworks, while having enough experience at this point to also pick up the smaller details while quickly being effective.
I feel like the comparison with Java is similar, as both Oracle and Microsoft are large corporations that largely control the language and ecosystem, while also having open source implementations. C# has made some more major changes to the language itself than Java, but both have diverged quite a bit from what they were back in 2005.
I'll find out next year, as my son goes into high school, and they are offering Java software development classes. I got Apple Basic on the Apple ][e, QuickBasic, and a bit of C++ in high school, and graduated in 1997. I wouldn't be surprised if he's going to be dealing with Java v1.5 instead of the latest features. At least I'll have a motivation to learn the latest features if he enjoys it and keeps working at it.
Well I certainly hope not… LTS for Java 5 probably hit end of life when he was 2. I’d say most likely the version being installed is Java 17, or 11 if theyre really dated. Java 8 would likely be the absolute minimum, mainly due to the industry being so slow to migrate away from it. Newer programmers are unlikely to ever know what Java felt like before generics and lambdas existed.
He's enjoyed playing with Lua for Roblox, and I've gotten him to go through some of the easier C# tutorials from Microsoft where the execution happens right in the browser.
> perhaps Erik Naggum, scourge of Usenet, was right when he said: “life is too long to know C++ well.”
I feel similar about Rust. I have read “the book”. Did a couple of small projects. It is also sort of committee administered language that sucks up tiny features like a giant vacuum cleaner sweeping the streets and changes every hour.
Can you provide some (preferably recent) examples? My experience has been the opposite. It feels like new features are given a lot of thought and deliberation, and stablizing even something small can take upwards of a year.
I will agree that today's Rust is different from Rust 1.0, but I don't see that as necessarily a bad thing. More that they were able to come to a stable 1.0-ready core fairly early on, and have been adding on the more tricky parts bit by bit since then.
C++ predates on C in a similar way to how Mojo predates on Python. At least C++ has extern C.
https://docs.modular.com/mojo/manual/python/#call-mojo-from-...
> As shown above, you can call out to Python modules from Mojo. However, there's currently no way to do the reverse—import Mojo modules from Python or call Mojo functions from Python.
One way street. Classic commons harvesting.
Regardless, I think it's a bit alarmist and overly aggressive to assume nefarious intent. Have the developers acted in ways such that this reputation is deserved?
Also, little OT, but it took me unreasonably long to understand that you meant "predates" as the verb form of "predator", not as in "comes before chronologically". The phrase "preys on" may be more clear.
> Regardless, I think it's a bit alarmist and overly aggressive to assume nefarious intent.
You can think those things. You can try and color my points using whatever language you want. That doesn't make it true, you are putting words in my mouth about "nefarious intent". Mojo is taking Python programmers and their programs out of the Python ecosystem and bring them into their ecosystem.
Thanks for the splaining about predates, I'll continue to use it. Mojo is definitely 'taking booty' from Python.
This depends on how they license it going forward, and whether they make it open, or use being a superset as a way to capture then trap python users in their ecosystem, and I don't think we have a certain answer which path they'll take yet.
The way they let you mix python compatible code with their similar but more performant code [1] looks interesting and provides a nice path for gradual migration and performance improvements. It looks like one of the ways they do this is by letting you define functions that only use typed variables which is something I would like to see make its way back to CPython someday (that is optionally enforcing typing in modules and getting some performance gains out of it).
[1] https://en.wikipedia.org/wiki/Mojo_(programming_language)#Pr...
This is already how Cython and MYPYC work. You add standard PEP-484 type annotations, and they use that to infer where code can be compiled to native instructions.
https://cython.readthedocs.io/en/latest/src/tutorial/pure.ht...
Mojo explicitly does the opposite, allowing Mojo to use Python but requiring Mojo to be in control, while making it hard/impossible for code written in Mojo to benefit code written in Python:
> Our long-term goal is to make Mojo a superset of Python (that is, to make Mojo compatible with existing Python programs). […] Mojo lets you import Python modules, call Python functions and interact with Python objects from Mojo code. […]
> As shown above, you can call out to Python modules from Mojo. However, there's currently no way to do the reverse—import Mojo modules from Python or call Mojo functions from Python. […]
> This pattern doesn't work because you can't pass Mojo callbacks to a Python module.
> Since Python can't call back into Mojo, one alternative is to have the Mojo application drive the event loop and poll for updates.
No comment on whether this should be viewed as an attack.
Same for Nuitka, and a bunch of transpilers
A superset of pure .py code, not the numpy, cython, ctypes and stuff.
But once you get "superset" of CPython's C bindings, congratulations, you get GIL.
I think the performance benefit would also be small. Many objects are only accessed by a single thread, some objects are accessed by many threads, but few objects are exclusively accessed by one thread and then exclusively accessed by a different thread.
if (op->ob_tid == _Py_ThreadId())
op->ob_ref_local = new_local;
else
atomic_add(&op->ob_ref_shared, 1 << _Py_SHARED_SHIFT);
I was a bit disheartened when the Unladen Swallow project [1] fizzled out. Great to see Python back on the core optimization track.
"tranced bread" is a fun name for some sort of library that breaks up files into pieces for better resilience for sending, like over BitTorrent.
I get in concept what the GIL is.
But what's the impact of this change?
Packages will now break, for the hope of better overall performance?
They assume they can "take the GIL", and then go and look at the various Python datastructures you passed them without worrying about them changing as they are being read.
The later, when writing out their answer (which might involve editing something they were given), they can assume they are not changing. For example, you could write code which extends the length of a list to 100, then fill in all the members, not worrying half way through your loop another thread shrinks the list back down.
Even outside of high-intensity CPU work, this can be useful. A problem lately is that a lot of code is written using Python's native asyncio language features. These run single-threaded with async/await to yield execution, much like in NodeJS, and can achieve pretty good throughput even with a single thread (thousands of reqs/second).
However, a big problem is that any time you do _any_ CPU work, you block all other coroutines, which causes all kinds of obscure issues and ruins your reqs/second. For example, you might see random IO timeouts in one coroutine which are actually caused by a totally different coroutine hogging the CPU for a bit. It can be very hard to get observability into why this is happening. asyncio provides a `asyncio.to_thread()` function [1] which can help to take blocking work off the main thread, but because of the GIL it doesn't truly allow the CPU-bound to avoid interfering with other coroutines.
[1] https://docs.python.org/3/library/asyncio-task.html#asyncio....
Edit: https://peps.python.org/pep-0703/ suggests it will support multiple cores, unless the current work does not yet achieve that.
I'm asking because I encountered a weird phenomenon before.
I use a simple Python lib called "schedule" which is to run some tasks periodically (not precise). And I often run a script multiple times (with different arguments) to monitor something say, every 30 seconds. So they're in three separate Python Interpreter processes.
What I've noticed is that while when I initiated them, they were something like 5 seconds apart, they eventually will end up running in sync. Probably not related to GIL at all, but I guess do no harm to ask.
The GIL only really kicks in if you use threads in a single process. Then, the GIL will only let one single thread do actual work at a time, and will trade off which thread gets to do work. The other threads can wait on IO stuff (web requests, the file system, etc) but they can't do number crunching or data processing at the same time.
That's a really interesting observation though, I wonder what _is_ causing your separate processes to sync up?
The OS scheduler has to execute M threads on N cpu cores, while also balancing competing priorities like latency and power usage.
Because each separate process uses a naive timer, the timings will drift slowly due to imprecision and small process scheduling delays etc.
After enough drift the timers will sync up by chance (harmonics), at which point OS scheduler incentives can lead the timings to “stick” together seemingly.
This may be a bit tangential, but I find it fascinating that there seems to be a mechanical version of this phenomenon with ancient roots - lookup spontaneous synchronization of pendulums.
By the way, I asked about it previously on their repo before, if you're interested.
No replies yet though, since the development of the lib isn't very active to begin with.
Intent to approve PEP 703: making the GIL optional - https://news.ycombinator.com/item?id=36913328 - July 2023 (499 comments)
Not if you use Windows, then it's a mess. I have a suspicion that people who say that the multiprocessing works just fine never had to seriously use Python on Windows.
* All Python webservers that somewhat support multiprocessing on Windows disable the IOCP asyncio event loop when using more than one process (because it breaks in random ways), so you're left with the slower select() event loop which doesn't support more than 512 connections.
1. Because global objects are refcounted, CoW effectively isn't a thing on Linux. They did add a way to avoid this [0], but you have to manually call it once your main imports are done.
2. On Mac, turns out a lot of the system libs aren't actually fork-safe [1]. Since these get imported inadvertently all the time, Python on Mac actually uses `spawn` [2] -- so it's roughly as slow as on Windows.
I haven't worked in Python in a couple years, but handling concurrency while supporting the major OSes was a goddamn mess and a half.
[0]: https://docs.python.org/3.12/library/gc.html#gc.freeze
[1]: https://bugs.python.org/issue33725
[2]: https://docs.python.org/3.12/library/multiprocessing.html#co...
I see this mentioned from time to time, but intuitively you'd think this wouldn't pose a big slowdown since the system builtin objects would have been allocated at the same time (startup) and densely located on smaller nr of pages. I guess if you have a lot of global state in your app it could be more significant.
Would also be interesting to see a benchmark using hugepages, you'd think this could solve remaining perf problems if they were due to large number of independent CoW page faults.
You can do whatever you want in the workers, I parse JSONs and write to sqlite files.
Once the Python ecosystem supports either subinterpreters or nogil, we'll happily migrate to those and get rid of our hacky interprocess code.
Subinterpreters with independent GILs, released with 3.12, theoretically solve our problems but practically are not yet usable, as none of Cython/pybind11/nanobind support them yet. In comparison, nogil feels like it'll be easier to support.
[1] https://docs.ray.io/en/latest/ray-core/walkthrough.html#call...
[2] https://docs.python.org/3/library/multiprocessing.html#manag...
Sharing arrays of numbers is supported in multiprocessing as well: https://docs.python.org/3/library/multiprocessing.html#shari...
Unfortunately, the closest thing we have to that is Julia, which fails to meet none of the requirements. Alas.
Julia’s threading API is really nice. One deficiency is that it can be tricky to maintain type stability across tasks / fetches.
I was intrigued by Julia a while ago, but didn't have time to properly learn it.
So just out of curiosity: what's the issues with jit and Julia ?
The famous "Time To First Plot" problem was about taking several minutes to do something like `using Plots; Plots.plot(sin)`.
But to be fair recent Julia releases improved a lot of it, the code above in Julia 1.10 takes 1.5s on my 3-year old laptop
I think Python is a terrible language that exemplifies the maxim "worse is better".
"My second [surprise] came a couple of hours into the project, when I noticed (allowing for pauses needed to look up new features in Programming Python) I was generating working code nearly as fast as I could type.
When you're writing working code nearly as fast as you can type and your misstep rate is near zero, it generally means you've achieved mastery of the language. But that didn't make sense, because it was still day one and I was regularly pausing to look up new language and library features!"
Source: https://www.linuxjournal.com/article/3882
It doesn't go for large code bases, but if you need quick results using existing well tested libraries, like in machine learning and data science, I think those statements are still valid.
Obviously not when you're multiprocessing, that is going to bite you in any language.
apart from go (maybe java) those are all "scary" languages that require a bunch of engineering to get to the point that you can prototype.
even then you can normally pybind the bits that are compute bound.
If Microsoft had been better back in the say, then c# should have been the goto language of choice. It has the best tradeoff of speed/handholding/rapid prototyping. Its also statically typed, unless you tell it to not be.
gets you 90% of the potential performance of a full multithreaded producer/consumer setup in C++. C++ isn't as scary as it used to be.
Of course, getting threads to be actually useful for concurrency (GIL removed) adds another very useful tool to the performance toolkit, so that is great.
Something that tripped me up when I last did `multiprocessing` was that communication between the processes requires marshaling all the data into a binary format to be unmarshaled on the other side; if you're dealing with 100s of MB of data or more, that can be quite some significant expense.
This topic is an example: a detail of one particular implementation, since GIL is definitely not inherent to the language. Just the usual worry about looseness of types?
The biggest impact would be completely redoing package discovery. Not in some straightforward sense of "what if PyPi showed you a Performance Measurement?" No, that's symptomatic of the same problem: harebrained and simplistic stuff for the masses.
But who's going to get rid of PyPi? Conda tried and it sucks, it doesn't change anything fundamental, they're too small and poor to matter.
Meta should run its own package index and focus on setuptools. This is a decision PyTorch has already taken, maybe the most exciting package in Python today, and for all the headaches that decision causes, look: torch "won," it is high performance Python with a vibrant high performance ecosystem.
These same problems exist in NPM too. It isn't an engineering or language problem. Poetry and Conda are not solutions, they're symptoms. There are already too many ideas. The ecosystem already has too much manic energy spread way too thinly.
Golang has "fixed" this problem as well as it could for non-commercial communities.
Or did you mean to say the "Python language"?
The "& derivatives" part is the problem! Torch does not have derivatives. It won. You just use it and its extensions, and you're done. That is what people use to do exciting stuff in Python.
It's the manic developers writing manic derivatives that make the Python ecosystem shitty. I mean I hate ragging on those guys, because they're really nice people who care a lot about X, but if only they could focus all their energy to work together! Python has like 20 ideas for accelerated computing. They all abruptly stopped mattering because of Torch. If the numba and numpy and scikit-learn and polars and pandas and... all those people, if they would focus on working on one package together, instead of reinventing the same thing over and over again - high level cross compilers or an HPC DSL or whatever, the ecosystem would be so much nicer and performance would be better.
This idea that it's a million little ideas incubating and flourishing, it's cheerful and aesthetically pleasing but it isn't the truth. CUDA has been around for a long time, and it was obviously the fastest per dollar & watt HPC approach throughout its whole lifetime, so most of those little flourishing ideas were DOA. They should have all focused on Torch from the beginning instead of getting caught up in little manic compiler projects. We have enough compilers and languages and DSLs. I don't want another DataFrame DSL!
I see this in new, influential Python projects made even now, in 2024. Library authors are always, constantly, reinventing the wheel because the development is driven by one person's manic energy more than anything else. Just go on GitHub and look how many packages are written by one person. GitHub & Git, PyPi are just not adequate ways to coordinate the energies of these manic developers on a single valuable task. They don't merge PRs, they stake out pleasing names on PyPi, and they complain relentlessly about other people's stuff. It's NIH syndrome on the 1m+ repository scale.
It is a non-optimizing bytecode interpreter and it makes no use of JIT compilation.
JavaScript with V8 or any other modern JIT JS engine runs circles around it.
Go, Java, and C# are an order of magnitude faster but they have type systems that make optimizing compilation much easier.
There's no language-inherent reason why Python can't be at least as fast as JavaScript.
JavaScript is just as monkey-patchable. You can reassign class methods at runtime. You can even reassign an object's prototype.
Existing Python JIT runtimes and compilers are already pretty fast.
In any case, that should be irrelevant to getting a reasonably performant JIT running. Lots of AOT and JIT compiled languages have robust FFI functionality.
The native extensions are more relevant when we talk about removing the GIL, since lots of Python code may call into non thread safe C extension code.
Working with threads is a pain in Python. If you want to spawn +10-20 threads in a process, it can quickly become way slower than running a single thread.
Removing the GIL and refactoring some of the core will unlock levels of concurrency that are currently not feasible with Python. And that's a great deal, in my opinion. Well worth the trouble they're going through.
Some might say: "Use Go!" Alas: https://songlh.github.io/paper/go-study.pdf
After a couple decades of coding, I can say that threading is better if it's tightly controlled, limited to usages of tight parallelism of an algorithm.
Where it doesn't work is in a generic worker pool where you need to put mutex locks around everything -- and then prod randomly deadlocks in ways the developer boxes can't recreate.
This may be a case of violent agreement, but there are a few clear cases where multithreading is easily viable. The best case is some sort of parallel-for construct, even if you include parallel reductions, although there may need to be some smarts around how to do the reduction (e.g., different methods for reduce-within-thread versus reduce-across-thread). You can extend this to heterogeneous parallel computations, a general, structured fork-join form of concurrency. But in both cases, you essentially have to forbid inter-thread communication between the fork and the join parameters. There's another case you might be able to make work, where you have a thread act as an internal server that runs all requests to completion before attempting to take on more work.
What the paper you link to is pointing out, in short, is that message passing doesn't necessarily free you from the burden of shared-mutable-state-is-bad concurrency. The underlying problem is largely that communication between different threads (or even tasks within a thread) can only safely occur at a limited number of safe slots, and any communication outside of that is risky, be it an atomic RMW access, a mutex lock, or waiting on a message in a channel.
That's not true at all. F#, Elixir, Erlang, LabVIEW, and several other languages make it very easy. Python makes it incredibly tough.
I disagree, Python makes it incredibly easy to work with threads in many different ways. It just doesn't make threads faster.
If I launch 50 threads with run away while loops in Python, it takes minutes to laumch and barely works after. I can run hundreds of thousands and even millions of runaway processes in Elixir/Erlang that launch very fast and processes keep chugging along just fine.
I'm not sure that argument helps your position on threading. I once saw a java program spin off 3000 threads doing god knows what. Debugging the fucking thing was impossible.
I think Java made it quite easy to spin off threads, and again, it doesn't help the argument. It just made the f'ing thing worse. Race conditions are still f'ing hard to solve. Particularly when a shared-mutable-state exists outside of the program.
Even if you have zero shared resources, zero mutexes, no communication whatsoever between threads, it's a huge pain in Python if you need +10-ish threads going. And many times the GIL is the bottleneck.
Like every other language I've used this approach with, nothing bad happened - the program ran as expected and produced correct results. Unlike every other language, spreading calculations across multiple cores didn't appreciably improve performance. In some cases, it got slower.
Eventually scrapped it all, and went with an approach closer to what I'd have done with C and fork() decades ago... Which, to Python's credit, was fairly painless and worked well. But it caught me off-guard, because with asyncio for IO-bound stuff, it didn't seem like threads really have much of a purpose in Python, other than to be a tripwire for unwary and overconfident folks like myself!
But now with async even that goes away.
as you know thats mostly threads in general. Any optimisation has a drawback so you need to choose wisely.
I once made a horror of a thing that synced S3 with another S3, but not quite object store. I needed to move millions of files, but on the S3 like store every metadata operation took 3 seconds.
So I started with async (pro tip: its never a good idea to use async. its basically gotos with two dimensions of surprise: 1 when the function returns, 2 when you get an exception ) I then moved to threads, which got a tiny bit extra performance, but much easier debugability. Then I moved to multiprocess pools of threads (fuck yeah super fast) but then I started hitting network IO limits.
So then I busted out to airflow like system with operators spawning 10 processes with 500 threads.
it wasnt very memory efficient, but it moved many thousands of files a second.
That said - I think it's fair to be irritated by people who write Python off as entirely useless because it is not _the fastest_ language. As you rightly say - it's fast enough for many purposes. It does bother me to see Python immediately counted out of discussions because of its speed when the app in question is extremely insensitive to speed.
I have been on teams where Python based approaches were discounted due to “speed” and “industry best practice” and then had the very same engineers create programs that are slow by design in a “fast” language and introduce needless complexity (and bugs) through “faster” database processes.
Like you said, it’s the thoughtless criticism. The meme. I am happy for Python to lose in a design analysis because it’s too slow for what we are building; I am loathe to let it lose because whoever is doing the analysis with me has heard it’s slow.
Which is to say, I get what you’re saying. I think people have been a little ungenerous with your comment.
Eh - I engaged with a fraught topic in a snarky way without clarifying that I meant the unintuitive-but-technically-literally-accurate interpretation of my words. Maybe some people have been less-generous than they could have been, but I don't begrudge it - if I look sufficiently like a troll, I won't complain when I get treated like one. Not everyone has the time and mental fortitude to treat everyone online with infinite patience and kindness - I know I sure don't.
Thank you for the support, though!
I see is people crying how python is slow and then use a proper fast programming language to write code that gets executed so few times that even if python was 100x slower it wouldn't matter or the program is so trivial that python's speed definitely isn't an issue.
I have even sometimes seen people stop using a tool when they find out they were written in python - now all of a sudden they are unusably slow. Then they try to justify it by writing some loop in their favourite proper fast language and tell me how fast that tight loop is or they claim that some function is X times faster, but when I actually compile it and run something like hyperfine on it and python version the difference is hardly ever X since there is already so much more over head in a real world.
I work on low-latency stuff and we routinely get server-side latencies in the order of single to low double-digit microseconds of latency.
If python ever becomes fully concurrent (python threads being free of any kind of GIL) we'll see the "python slow" meme for a number of years... Also doesn't help that python gets updated very very slowly in the industry (although things are getting better).
Of course, if you have a simple fastpath you can make it fast in any language with a JIT, latency is also generally not an issue anymore, credit where credit is due - java GCs are light years ahead of everything else.
Regarding jlink - my main complaint is that everything requires java.base which already is 175M. And thats not counting the VM, etc. But I don't actively work with java anymore so please correct me if there is a way to get smaller images.
Probably the right design though.
AFAIK we're just talking about removing the global interpreter lock. I'm pretty sure the threading library uses system threads. So running without the GIL means actual parallelism across system threads with shared memory access.
EDIT:
> [the test synchronous programs] all seem to run fine, and very basic threaded programs work, sometimes
Perhaps this is closer to removing the oil pan
> When we found out about the “nogil” fork of Python it took a single person less than half a working day to adjust the codebase to use this fork and the results were astonishing. Now we can focus on data acquisition system development rather than fine-tuning data exchange algorithms.
Read PEP-703 (https://peps.python.org/pep-0703/#performance) where the performance hit is currently 5-8%
If you only care about single thread there's all kinds of stuff you can do.
Async I/O and threads are two different things, and either can be present in real code without the other.
I have ran a lot of programs containing race conditions successfully many times until I ran into an issue.
At any rate, test_asyncio contains a lot of tests that involve threads and specifically thread safety between coroutines and those tests fail. As far as async I/O and threads being distinct, I mean sure that is true of a lot of features but people mix features together and mixing asyncio with threads will not work with this particular release.
> small threaded programs had been run successfully
The second obviously contradicts the first, doesn't it?
> mixing asyncio with threads will not work with this particular release.
That's a very different claim to the first, and one that no longer conflicts with the second, isn't it?
You're not supposed to drive a car that hasn't got out of the research and development laboratory either, so there's that.
Only as long at it’s as easy to put back in
I also want types, so Elixir is not in the picture. I dabbled in Rust a bit. Although I was able to get the hang of things and build a CLI tool pretty quickly, I'm worried I'll have to deal with numerous quirks later if I keep using Rust (like numerous string types). Is that something to be worried about if all I want from Rust is Python+Types+Concurrency?
You can even run Python from Julia, so that alleviates the problem with a lack of libraries somewhat.
It absolutely is - it's just testing at a _very_ low level of correctness, and is not sufficient for testing actual high-level functionality.
Sounds like Groovy. But I wouldn't recommend it. Also the career padding hype is gone.
But once you get over it, you realize Golang has a good type system, concurrency model, package manager that's not pip, fast compile times, and static binaries. For most cases it will also offer great performance.
It has everything you need to build APIs, CLI tools, web servers, microservices - pieces which will form the building blocks of your software infrastructure. I have heard numerous stories of people being productive in Go in a few days, sometimes even hours.
If Python is 0 quality of life and Rust is a 100, Golang gets you all they way up to 80-90. That last bit is something you might never need.
Rust is a great language and something I hope to be in proficient someday, but I ll save it for where I actually need that last microsecond of performance.
If you're not a fan of GoLang's spartan syntax and are cool with async/await, C#/dotnet core is a great experience on all platforms. IMO it has the best async/await implementation (it originated there) on top of a multi-threaded event loop. ASP.NET is a great web framework and it has great library support for everything else. As someone who avoids "traditional" ORMs (Hybernate, Django) I really like Dapper.
Nim is a statically-typed compiled language with very pythonic syntax. It's easy to learn, especially if you already know Python, because Nim's stdlib is heavily inspired by it.
For multithreading in Nim see:
Weave - https://github.com/mratsim/weave
Malebolgia - https://github.com/Araq/malebolgia
[0] - https://nim-lang.org
I migrated a couple of projects from Java, TS and Go to Rust and honesly, I couldn't be happier.
Learn golang or rust instead. Impressive that they are managing this though!
>> 99% of my compute is offloaded to compiled BLAS or CUDA.
types are enough.
and memory and type safety are not terribly relevant for my use cases.
With tools like pyright now + the work on nogil everyone benefits from this using Python.
Also Rust became a quite popular tool for python extensions, where you can offload performance to rust and business logic to python.
Yes it is, in every way. When the first job of choosing to use types is choosing which of the five or so type checkers, each deficient in their own way and incapable of dealing with the poor idioms the language encourages to proliferate, you know you're on the wrong path.
Bad take. Learn golang and rust and python.
You should use the language which is suited to the task, sometimes that's golang sometimes that's python and sometimes it's rust.
It's impressive that the python team as a whole continues to improve in such big ways after more than 30 years of development. It's more impressive that the python team managed to navigate 2to3 and come out stronger.
Having to "if err != nil" every single function call is a big put off - imagine having to "try catch" everything in a language like C#!
It forces me to think about how my programs can fail and what I should do when they fail.
func Must(err error) {
if err != nil {
panic(err)
}
}
func Panic[T any](v T, err error) T {
if err != nil {
panic(err)
}
return v
}Instead, pick C#/F#, Kotlin/Clojure or Rust depending on the use case.
I definitely agree that libraries can and should drive your language decisions. A 20,000 line golang program might be 10 lines of python because there is a library to do what you need. Similarly a complicated-to-reason-about python program may be made far simpler by using go channels/routines.