Analyzing Data 170,000x Faster with Python
sidsite.com
sidsite.com
The rust code is pretty simple, and plain, and doesn’t need any of these tricks
It’s very readable to me (not a data scientist, but I’m decent at Python), YMMV.
I don’t think most people need this level of accuracy, but it is something to consider.
Database queries are even worse, because not only do you need representative data at a similar scale (or you’ve scaled both hardware and working set accurately), but you have to hope that the table/column stats in prod and test are kept at a similar level of accuracy, which is more art than science.
Did the query regress because of the change, or because writes to an unrelated table have increased load and the analysis daemon is now struggling? Did it regress in test because someone else pushed a change to a rarely-used query, causing buffer churn when CI ran your job? Etc.
If there’s a CLI tool written in Python, ideally install it with pipx.
Use Poetry for every project.
Unfortunately these aren’t inherently obvious until you have a raging tire fire, and even then, you have to know the source of it and that better options exist.
But hey, at least it isn’t Node.
I still have hopes that Julia and Ruby competition makes it change.
In practice, I have seen Node apps blocking or crashing far more often than Python apps, which drags the performance down just a bit.
In the meantime, @njit decorator on Python functions, and boom, done.
@njit doesn't work in regular Python, requires installing Numba and only works in very specific use cases.
As to specificity, I’d argue that it works on the things which stand to benefit the most from it. Parallelizing loops, vectorizing, etc.
The numba jit is for numerical code. You won’t magically get better performance iterating over a bunch of things you get back from your db for example.
Why? Node's packaging story is about a million times better than Python's. You literally just listed some of the issues with Python, none of which apply to Node.
> use venvs liberally
Not necessary with Node (you could argue it always uses venvs but that's transparent from a user point of view).
> to let as few things as possible interact with system Python.
Not an issue.
> If there’s a CLI tool written in Python, ideally install it with pipx.
Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job).
> Use Poetry for every project.
You don't have to magically know which project management tool out of dozens to use with Node. Just use the overwhelmingly popular one (NPM).
You missed a few more by the way. How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling? Oh and remember that that is deprecated with no alternative! And it silently doesn't work in `requirements.txt`! Yeay.
Then there's `pip`'s fun "wheel error! this probably isn't a wheel error" message. Very user friendly.
Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows.
Node is a million miles ahead of Python in terms of packaging usability.
No, instead you get to deal with absurdly slow dependency solving / installation (I gather Bun is fast, so good on that guy), the existence of leftpad, and a `node_modules` directory that is stupidly large.
> Not an issue. [to let as few things as possible interact with system Python.]
Obviously, because systems aren't issuing tooling in Node.
> Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job).
This is literally untrue – I know for a fact that both npm and yarn are extremely popular, and as mentioned, the newcomer bun exists.
> How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling?
Or, you know, switch to modern solutions for this: `poetry add --path /foo/package` or `pipenv install -e /foo/package`.
> Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows.
Don't use Windows, obvs. Seriously though, I give absolutely zero fucks about development experience on Windows with Python.
…have you installed anything with Python recently? It’s not exactly speedy itself. Oh and who knows whether said application will do the right thing and use a venv or just spew its packages into the global path. At least node has the dignity to put all its project-scoped dependencies into a single place, separate from everything else.
> Obviously, because systems aren't issuing tooling in Node.
Given I can’t remember the last time I used an application that relied on the system providing the runtime (let alone the correct version) instead of shipping their required version, I’m not sure this is really much of a point.
> This is literally untrue – I know for a fact that both npm and yarn are extremely popular, and as mentioned, the newcomer bun exists.
And all of them are more streamlined than your average Python packaging experience, I promise. I could probably get my parents to NPM install a working with greater success and ease than trying to get them to do the same with whatever is the latest Python packaging mechanism de jour.
> have you installed anything with Python recently? It’s not exactly speedy itself
Yes? I’ve never thought, “man this is slow,” which is very unlike my few experiences with npm. IIRC yarn was quite a bit faster, but I haven’t used either in some time. If you have a list of packages as a benchmark, I’m happy to try to come up with an equivalently-sized Python list and compare the two.
> Given I can’t remember the last time I used an application that relied on the system providing the runtime (let alone the correct version) instead of shipping their required version, I’m not sure this is really much of a point.
I think you’re missing the point – systems often ship with tooling written in Python because it’s a reasonable language, with a relatively stable ecosystem. Python versions don’t EOL for at least 5 years. Node is about half that.
NPM is definitely faster than pip to install things. Not sure what you're talking about here.
> I know for a fact that both npm and yarn are extremely popular
The point is they both work. Nobody is going to say "you picked the wrong package manager out of these 8 options". There's an obvious option and a slightly less obvious option and they both work properly.
> switch to modern solutions for this
What was I just saying about having to magically know the "right" way to do it? In any case I don't think either of those will solve the issue.
> Don't use Windows, obvs. Seriously though, I give absolutely zero fucks about development experience on Windows with Python.
Sounds like you give zero fucks about the Python development experience on Linux too, otherwise you wouldn't defend its obvious deficiencies.
Most recently it defaults to asking for a keyring, even when just downloading? I believe that's marked WONTFIX. Even with that hacked around, and I thought the point was to reduce the number of hacks required, tensorflow seems to almost universally fail in absolutely any configuration I've tried.
Don't even begin to try integrating your own packages, c++ template errors are tame by comparison. I've also never seen a poetry error fit in less than 500 lines of nested exceptions.
I want to like this tool but... boy, Docker doesn't give me this crap.
Poetry is tightly optimised for what they consider the 'correct' way of developing 'normal' python apps. Wander off that path and poetry becomes worse and worse.
It also fits with the Zen of Python (which tbf rarely seems to happen) of, “There should be one-- and preferably only one --obvious way to do it.”
Node / JS in general seems to take the path of, “this existing solution doesn’t precisely fit my needs, so instead of rethinking why that might be, I’m going to reinvent the wheel.”
Given that you where arguing that using python on the most popular OS in the world should fall outside of a 'normal' use case python should focus on, perhaps you have a too narrow view of what "correct and normal" python usage should be.
No they're not complying to your CI/CD best standards - tell me with a straight face they're learnable in less than a month with a budget of $0 above your existing capital. Yes they're adding real value to the economy. That's "computing for everyone", and one of those tools works for that scenario.
I think the real takeaway from my experience of Rust vs Python performance is that in practice people don't optimise 99% of Python so the real number of significance is the 8x speedup he got from just switching to Rust with no profiling whatsoever.
That's actually quite a low difference compared to my experience - I'd say 50x is more typical.
As it stands, any AOT/JIT compiled language has an edge over Python, considering using only the reference implementation, which is quite relevant on systems were one cannot expect to allowed to install additional development tools.
As for the bindings to native libraries, yeah Python might be favoured for bindings, however those native libraries can be called by any language that has FFI capabilities anyway, alongside their JIT/AOT capabilities.