Hacker News Reader

Comment by ZoomerCretin | Hacker News Reader

Would the authors not have been better off adding their performance improvements to pip rather than creating their own tool? I remember this being a complaint from Node maintainers when Bun was released: they had multiple open performance-related issues with no one available to work on them.

View on HN

Much like how people underestimate the time and effort it takes to build mature software from scratch, people also seem to under-estimate greatly the complexity and effort involved in trying to make huge improvements to existing tools, both technical and political in nature. There's absolutely no chance that the vision the Bun folks had could be carried out in the Node.JS project in my opinion, the resulting attempt would have less impressive results and probably would leave everyone unsatisfied. One of the biggest benefits and most powerful features of open source is undoubtedly the fork, but just as important is knowing when not to do that, and that's definitely the case when the position you want to be is more different from any existing starting point than it is from scratch; I'd argue writing a 'fast' Pip-compatible package manager from first-principles is a perfect case for that.

Re: politics. Note that sometimes forks and competitors are so successful, they make it possible for the upstream to "unblock" by virtue of convincing them to adjust their vision. For Node.JS, consider io.js and Yarn in particular.

I’m guessing “rewrite it in Rust” would have been an unpalatable PR for the pip maintainers to merge.

Yeah "lets rewrite in Rust" has become a meme with negative connotations that will attract ridiculed or be ignored, but Rust clearly excels in this sort of tooling. Whether it makes sense to rewrite an existing codebase or pigeonhole it on the edges is another question.

Yeah but I suspect there’s something more going on than Rust magic. They’re claiming a savings of about an hour and a half. (4% of the original time) That sounds like an architecture change, or more likely a whole lot of caching.

I guess the uv authors don't want to maintain a better algorithm in Python then

I don’t know how you can seriously imply that Python could be just as fast as Rust if the authors of uv cared enough.

That seems like a disingenuous framing of why UV exists, let alone doesn’t pass the smell test that Python can just be as fast as a compiled, native machine code language. Python and Ruby and all that are just so god awful slow that it’s completely obvious that’s a rewrite in native code would have these sorts of benefits.

If the first step of your build process is creating a new pip venv from scratch, it's entirely believable to me. Pip resolves packages sequentially, so there's a lot of low hanging fruit even without yoinking a state of the art constraint solver.

As a concrete example, I have a raspberry pi 4 I use for "what's the most pathetic hardware a user might try". It takes 5-10 minutes to build up a fresh venv even when it has all of the packages cached. If I nuke the caches and make it download, it's 20+ minutes. For comparison a modern laptop takes ~1 minute.

Is that on a SD card? My expectation is that SD cards are garbage and any number of IO access patterns break down vs a SSD.

I threw in a moderately fast USB drive (~90 mb/s after pi4 overhead) because I got sick of SD cards dying every 6-12 months. Pi4 is very speed limited. Even if you tried to use an ssd, I think the highest I've seen is around 250mb/s with an nvme drive that gets a few gb/s in a real computer. In theory it should get to like 500mb/s for sata limits.

Your assignment, should you choose to accept it, is to write a small, simple Python program. Maybe something that builds a list of five or six elements, runs a simple map over it, then converts it to a dict somehow. Something simple.

Then run it through gdb and watch every single CPU opcode that is executed for the code you wrote. Not Python opcode; CPU. Print something first and trigger off of that or something, I'm not worried about Python startup costs here.

Then do the same with a roughly equivalent compiled program. Doesn't have to be Rust, C or whatever you know would do just fine.

Yes. You can get a big speed improvement over pure Python simply by moving it to a compiled language and making effectively no other changes. Python is slow. This is not a value judgment. It is a big mistake to think that this is somehow an emotional judgment about Python stemming from anger or hatred or something and that anyone talking about Python's speed issues are just haters or something. I like Python just fine. But you will understand how it is not a value judgment when you are running the Python program through gdb. It is simply an engineering reality that anyone using Python needs to understand about Python. You can get yourself into a lot of trouble not understanding this about Python.

Is there more things going on than a straight port? Possibly. But I don't find the difference unbelievable for it to be coming from a straight port with no major changes.

Converting python code to rust for data processing can get you 20x perf increase (single threaded).

Most innovations happens outside large organizations, then either replaces it or the ideas get stolen/absorbed into the parent. All the downsides and barriers of operating in a large legacy system are pretty well known, even if the maintainers are receptive, radical change will be DOA.

It's tricky to change up some parts of pip because it might break some existing users flows. It's really hard to test for this kind of infrastructure tool because it's so environment-dependent. The rewrite lets there be a clean slate. Meanwhile the target of being "compatible" with pip gives them a good idea of where to go, and they can focus a bit more on testing given the lessons learned.

This is definitely a paid ad. Check out the company cited in the post. It's a VC funded attempt to extend/embrace/extinguish the current tooling. [0]

https://astral.sh/about

Yes, and Linux was an attempt to extend/embrace/extinguish Unix.

You can't embrace/extend/extinguish OSS with other OSS. I have my doubts about Astral, but the software they make is OSS.

Astral is a company but that doesn’t make this an ad without evidence you didn’t include. Do you personally know that this was anything other than organic praise from a happy user?

Happy to report that this definitely isn't a paid ad. The Home Assistant team did this on their own. I helped out by building uv for some of the architectures they needed: https://github.com/astral-sh/uv/pull/2417

uv isn't a performance improvement on pip but a ground-up rewrite in a completely different language.

Source on the logic being different from pip?

Last I checked it's nothing more than a port of pip to rust.

I’m unclear on how you “port” a library from Python to Rust, it’s a ground up rewrite with a familiar interface to pip but fundamentally doesn’t share any code.

It's actually pretty easy to call python from rust and rust from python. An incremental port of a python library into rust would definitely be possible.

I'm not sure it would be a good idea. Python patterns don't necessarily make good rust patterns. Even ignoring larger code structure (e.g. single ownership*) you'd practically have to do a second pass to stop using python compatible types (except in the exposed interface) if you wanted an idiomatic rust library at the end of it. But it's definitely possible.

* Which you don't "have to" do in rust, you can use `Rc<RefCell>` everywhere and imitate python instead. It's just throwing some of the advantages of rust and adding some noise to type signatures.

It is not.

> Limitations

> While uv supports a large subset of the pip interface, it does not support the entire feature set. In some cases, those differences are intentional; in others, they're a result of uv's early stage of development.

> For details, see our pip compatibility guide[1]

> Like pip-compile, uv generates a platform-specific requirements.txt file (unlike, e.g., poetry and pdm, which generate platform-agnostic poetry.lock and pdm.lock files). As such, uv's requirements.txt files may not be portable across platforms and Python versions.

[1] - https://github.com/astral-sh/uv/blob/main/PIP_COMPATIBILITY....

both can be true.

i think now their current positioning is that the competition actually helped demonstrate the value of some kinds of speed, and that finally got those performance issues prioritized enough to work on by people with the right knowledge. Node doesn't lack for people who want to contribute, its more about people who -can- contribute well, and yes as other comment noted, its often to easier to try things out outside the system rather than work within it.

Reply on news.ycombinator.com