Hacker News Reader

Comment by danpalmer | Hacker News Reader

I'm a little sad that Ruff took off as a re-implementation of a whole bunch of work that was already done, rather than as a project to improve the work that was already done.

It was nice to be able to write little extra linters or flake8 plugins in the language I was linting. Plus decades of effort had gone into making those plugins pretty great, finding the right separation of linting codes so that the right things could be linted/ignored per codebase.

I understand why they did it, and they have done it remarkably well, but "rewrite it from scratch" is almost never the best answer and part of me wonders if this result could have been even better achieved another way.

View on HN

I'm also sad but from a different perspective.

If something can be rewritten from scratch fairly quickly and ends up being much faster, it makes you wonder what we (the general “we”) might have done “wrong” with the “decades of effort” before.

To be clear: there was nothing wrong with the old toolchains (hence the quotes). And of course, Ruff/uv can do what they do because they build on all that earlier knowledge. My point is just that sometimes it's easier to start over than to fix the old, and that fact itself feels kind of sad to me.

> If something can be rewritten from scratch fairly quickly and ends up being much faster, it makes you wonder what we (the general “we”) might have done “wrong” with the “decades of effort” before.

I think the impact of Rust (and Go) shouldn't be underestimated here. Prior to these languages if you wanted fast runtime you were either using C or C++ with all their associated footguns, tricky build systems, and poor support for exposing and consuming libraries. Or you were using something like Java or C# which meant managing another runtime just for tools, which is especially bad for something like UV which manages runtimes - imagine if you were using a python tool to manage your Java versions and a Java tool to manage you Python versions, it would be a mess!

With both Go and Rust you have fast languages without memory safety issues that compile to a single, easily-installable binary and have a broad ecosystem of libraries available that mean that half the time you don't even need to write any difficult code and can just glue some modules together, and the other half of the time you only need to write the code you actually want to write and not all the support code for general pupose stuff like reading config files, file watching, etc.

> there was nothing wrong with the old toolchains (hence the quotes). And of course, Ruff/uv can do what they do because they build on all that earlier knowledge

I don't think that's true. I think the old toolchains really were bad, partly because the Python community was uniquely resistant to building on what worked in other ecosystems, and uv in particular largely succeeds because it's the first tool in this space to finally just ignore the nonsense that had somehow become accepted wisdom in Python-land and apply a little bit of outside experience.

> I think the old toolchains really were bad, partly because the Python community was uniquely resistant to building on what worked in other ecosystems

I mostly attribute it to backwards compatibility concerns (which I in turn attribute to the traumatic 3.x migration). PyPI continued to accept egg uploads until August 2023 (https://packaging.python.org/en/latest/discussions/package-f...). The easy_install program, along the direct command-line use of `setup.py` (as opposed to having Setuptools invoke it behind the scenes), have been deprecated since October 2021 (https://setuptools.pypa.io/en/latest/history.html#v58-3-0); but not only are they still supported, Setuptools can't even remove support for `setup.py test` (as they tried to in 72.0 and immediately reverted - https://github.com/pypa/setuptools/issues/4519) without causing major disruption to the ecosystem. One of the packages affected was Requests, which: a) is one of the most downloaded packages on PyPI; b) is pure Python (aside from dependencies) with nothing particular complicated about its metadata (I see no reason it couldn't be done in pyproject.toml and/or setup.cfg); c) wasn't even reliant on that interface to run tests (they use Tox).

(For that matter: Requests has been maintained by the PSF since mid-2019 (https://www.reddit.com/r/Python/comments/cgtp87/seems_like_r...) and the problem could have easily been avoided once the deprecation was announced, but nobody did anything about it. The project still defines most of its metadata in `setup.py`; `pyproject.toml` exists but is only used for pytest and isort config, while `setup.cfg` is used for flake8 config and some requirements metadata that's completely redundant with `setup.py`.)

A lot of it also just has to do with lack of attention and focus. https://peps.python.org/pep-0427/ (defining the wheel format) was proposed in September 2012, and accepted in February 2013. But Setuptools itself wasn't available as a wheel until November 2013 (https://setuptools.pypa.io/en/latest/history.html#id1646), and in 2017 there were still reports of people having outdated versions of Pip and not being able to install Setuptools from that wheel (https://setuptools.pypa.io/en/latest/history.html#v34-0-0). Setuptools relied on a separate package to actually make wheels until July of last year (https://setuptools.pypa.io/en/latest/history.html#v70-1-0) - an effort which was initially proposed in June 2018 (https://github.com/pypa/setuptools/issues/1386). It also took years to notice that Pip and Setuptools had separate implementations for the code that understands the "tags" in wheel filenames and factor it out into `packaging` (https://github.com/pypa/packaging/pull/156).

> Requests

I hate to say it, but the development of Requests itself has stagnated and really needs some attention.

The most notable fiasco recently was the introduction of significant changes to TLS/SSL in version 2.32.0 [1][2], which caused widespread breaking issues and even led to a security vulnerability.

Attempts to address these problems in versions .2 and .3 introduced new major issues which still exists in current version [3].

A patch to resolve the new issue was provided by one of the core members of the Requests project as early as June 2024 [4], but for some reason, nothing has been done about it, despite repeated pushes from the community.

If you check the commit history, updates have been sparse lately, even though there are still many unresolved issues.

[1] https://github.com/psf/requests/issues/6655

[2] https://github.com/psf/requests/pull/6667

[3] https://github.com/psf/requests/issues/6730

[4] https://github.com/psf/requests/pull/6731

I've mostly ditched requests in favour of httpx these days. https://www.python-httpx.org

> My point is just that sometimes it's easier to start over than to fix the old, and that fact itself feels kind of sad to me.

Sad, but also liberating. There are surely more projects out there that would benefit from breaking with conventional wisdom to start over.

I am quite grateful that they rewrote so many tools that I had been using. Upgrades were super painful with them split into a dozen separate packages, I routinely ran into incompatibilities and had to pin to specific versions to deal with transitive dependencies.

Given the state of python packaging and tooling, I'd say that consolidating the tooling is a big win in and of itself, and with the tremendous speed wins on top...

Whenever Ruff comes up, I reflexively go and check https://github.com/astral-sh/ruff/issues/970, since Pylint is the one tool whose performance I find to be egregiously bad. Looks like Ruff is still very far from being able to replace it.

What about ruff + pyright? From a skim it seems like a lot of the missing features would be covered by a type checker?

(Pyright isn't written in Rust, but adequate performance is still a design goal)

Rewrite from scratch is exactly what the Python ecosystem needs more of.

The strong push to always contribute to the existing projects rather than starting new ones in the Python community is what caused the previous subpar state of tooling. I think Astral is also picking a good middle-ground where they still aim to build as much as possible on existing Python standards (and help contributing to new ones), so it's still building on what has been done in the past.

There was probably a lot of engineering that went into those designs and that took time.

Having a clean process to rewrite from is what made it so fast. They knew exactly what outcome they wanted and they had access to the internal implementations on how to do it.

All that effort was not wasted at all.

> this result could have been even better achieved another way.

Don't you have "decades of effort" to resolve this wonder?

Your argument seems ideological. There is no chance they could have improved Flake8 to be as good as Ruff is.

Folks said the same thing to me about grep 8.5 years ago when I released ripgrep.

FWIW, I think the reason I'm conflicted is probably a similar reason to why you made a separate new thing. Overcoming the inertia and ways of doing things, or proposing widespread changes, often doesn't go down well with existing maintainers – for very valid reasons. I probably wouldn't want to go into the grep project and suggest rearchitecting it, and I can see why the Ruff developers didn't want to go into projects like Flake8 and Pylint to do the same.

But that doesn't stop me from feeling that there were things lost in this process and wishing for a better outcome.

A huge advantage of going your own way is that you don't need to address an audience you think are just wrong.

When this happens periodically (rather than, every other week) you also get the original GNU advantage which is you can revisit an old tool but knowing what you know now, for example today "everybody" has DVCS, and so ripgrep checks for the .gitignore file.

Wait... Did you misunderstand this comment? Or are you saying grep caught up with ripgrep now?

I'm agreeing with them. Some folks told me I should have improved grep instead of building my own thing.

And you think you've done it?

Yeah it's somewhat ideological, I think open source software is better for society when built as a community project than built and controlled by a VC funded company.

I don't think you could get flake8 to be as fast as Ruff, but I think you could improve it to be within an order of magnitude, and honestly that's plenty good enough. There are a lot of low hanging fruit, particularly around caching. I'd push back on there being no chance of being "as good as" Ruff, because "good" is not just speed. Ruff is _not complete_, if you want all the features and linters, you still need Flake8, or you have to sacrifice some. It's also not publicly extensible, and not written in Python, both of which are fair choices but I think deviate from the ideal scenario here.

Blame Guido. Until recently when he was bought out by Microsoft, he had been the primary blocker of higher Python performance. There were a bunch of attempts at adding a JIT to Python but Guido was more interested in splitting hairs over syntax than any real heavy lifting. Python could have been as performant as LuaJIT or V8 but their dictator ruined it. Python need more Mike Palls.

Flake8 and Black would never have matched the performance of a Rust-based tool, even if they had 10 years of dedicated focus.

Reply on news.ycombinator.com