We Rewrote Our Project with Rust and It’s Almost 40X Faster | Hacker News Reader

We Rewrote Our Project with Rust and It’s Almost 40X Faster

betterprogramming.pub

37 pointspeefy14 comments

Sure, Rust is fast, but Python is really, aggressively slow. They could rewrite in basically any compiled language (or even some of the dynamic ones) and come out way ahead. It's such a frustrating experience trying to wring halfway decent performance out of a project written in Python when the lowest level fundamental operations like "accessing a property on an object" and "looping" are performance landmines.

>It's such a frustrating experience trying to wring halfway decent performance out of a project written in Python

It's something I've noticed as well and it kind of makes me sad because I actually like Python a lot. But I recently wrote something in Clojure, which is still dynamic and high level, and it was like 15x faster without me needing to think much.

It's the same with Python's async story. Nothing has made me ever feel more stupid than trying to learn asyncio.

With the help of Pyo3, the interoperability between Rust and Python becomes easier. We use Rust to write the performance-sensitive parts and bring the capabilities to Python through binding. The performance-sensitive parts can still be written in Python. After all, Python's ecosystem is slightly richer than that of Rust

Those are not the fundamental primitives of python programming. The base primitives are numpy, sqlite, gstreamer, and whatever other high level blocks exist as an extension module.

Python can be fast as long as you think in terms of "low level doesn't exist, if I'm looping over items myself there's a problem", right up until you have a problem that can't be expressed in terms of very high level blocks.

I suspect they will be faster than other compiled languages though. The safety will make people less afraid to optimize, in C it's pretty much keep it simple or you're screwed, rust actually helps manage complexity instead of seeming to declaring complexity as not a well supported use case.

I'm sure there's some truth to what you're saying here; chiefly that there are some well optimized libraries out there that handle certain types of problems very well, but the comment reads as satire. A fundamental primitive should be built in, provided as the default, and high performance.

Recently I was troubleshooting why a protocol parsing driver we have written in python was performing so far under benchmarks. After hours of debugging I found that an enum had been added to keep track of a specific state, and that it had increased the processing time by 100 times! How am I supposed to trust a language that can't even get the basic data structures right!?

Enums being that slow sounds like something they will probably address at some point.

But in general the whole point of Python is that you rarely need stuff like protocol parsing drivers, you just use premade bytestream parsing engines that have Python bindings if you can, although that's not always practical.

With Python rather than asking "How do I implement this feature from first principles" you can ask "How do I implement this in terms of these large high level blocks that might or might not actually be what the problem logically calls for, but nonetheless they are blazing fast and can do the job, even with the stuff you have to do to adapt them for your task, or do I have to write my own Rust extension".

Of course that doesn't help much with uniformly slow web code, which I would imagine is why Python isn't #1 for web backends, along with typescript being really awesome these days.

Frankly, they could have and had a massive amount of refactor work... Rust and python actually are syntactically similar enough that this conversion is the easiest... I know I have done it.

Here’s a classic language benchmark:

https://devm.io/java/energy-efficient-programming-languages-...

That article just cites the original research. Obviously it depends on exactly what they were measuring, but Rust is technically one of the fastest languages.

Personally I think another comment’s point is important too: I’ve always thought of Rust as Pythonic-C, so I’m sure the translation is easier.

Rust does learn from python, such as crate and module, to some extent. The compilation tool of Rustc itself is written by python, interesting.

Basic context seems like an ideal opportunity for a rewrite. Other benefits besides speedup in the article.

> The project we are developing using Rust is called KCL. (KCL) is an open-source constraint-based record and functional language. It improves the writing of complex configurations through a mature programming language stack.

> KCL was written in Python before.

> We used Python, Go, and Rust to implement a simple programming language stack virtual machine and compared their performance. In this scenario, Go and Rust have similar performance, while Python showed a significant performance gap. After careful consideration, we chose Rust.

Would have been interesting to hear why they didn't benchmark other languages suited to programming language implementation. Go is obviously rather weak in expressivity there, and other options which have both automatic memory management and high performance exist.

But the 66% bottom line performance improvement does sound like a nice improvement.

Supplement: Although we did not use Go and Python to write the core compiler part of KCL again, we provided KCLVM Go and Python SDK for peripheral ecological supplement: https://kcl-lang.io/docs/reference/xlang-api/overview

We rewrote our project with <insert compiled language> and it's almost <insert number>X faster

Well, than Python. Who would have thought that?

Reply on news.ycombinator.com