Hacker News Reader

The main reason pylint is slow is it tries to infer the types of the variables, and it predates type annotations so it can go through multiple functions/control paths across multiple files to figure these out. This approach is obviated by type annotations and type checkers.

Ruff is built for speed from the start and doesn't look at type annotations because you should run a type checker alongside.

Just to be sure. Is Ruff faster by lacking type checking?

Also, does Ruff work well on codebase without type hints?

Ruff is definitely faster by lacking type-checking. That requires way more analysis.

Ruff will work fine without type hints.

I believe it rightfully leaves it to mypy for those who want those features.

Mypy transpiles itself to c using mypyc and that can still take a while to complete when caches get invalidated.

Ruff is faster than many tools which don't do type checking.

Ruff works well regardless of whether you use type hints.

that's dodging the question a little bit? let's say Ruff is faster because it's so good, how much faster than type checking alternatives would this great tool be if it did type checking?

Without building the tool to compare, it's really hard to tell. In my experience of rewriting some Python and Ruby code into compiled versions, you can expect 10-20x improvements. But it's both really dependant on the code itself and on how you're going to reimplement it - most of the time it doesn't make sense to do perfect 1:1 rewrites.

That depends on how you do type checking and more specifically type inference. Mypy infers types of local variables but not function boundaries as it expects you to annotate function arguments and return values.

In mypy a variable taking values of two types is an error (unless explicitly annotated) which means you can assume the type from the first assignment statement and then use that assumption every other time the variable is used/assigned. In pylint it allows variables to take multiple types and only emits errors when an operation is invalid against one of the types. E.g. `x[0]` is valid for lists and tuples so you could assign it a list or a tuple depending on some condition.

The number of possibilities it considers quickly multiplies. Eg if your function is of the form `if blue_condition: x = blue_action(x)` that's a doubling of the possibilities and ten flags -> 1024 possibilities. (pylint has some heuristics on when to give up.)

The advantage of pylint is that it doesn't require your code to fit a certain style. But nowadays people have type annotations and they prefer to use them even if it means changing their code style a bit here and there.

Other type inferers with different approaches are pyright, Jedi, Pyre, pytype etc with different tradeoffs.

Even if no VC was involved, how do you make money developing an open source linter?

They are probably looking for an ecosystem around the tool, like how github built on git.

How did Hashi corp make money (and an eventual billion dollar plus IPO!) developing open source tools like vagrant, consul and terraform...

Clearly the creator of ruff is going to expand from an open source tool into a company focusing on Python tooling and developer experience, services, etc. There is a market for stuff like that just based on pycharm and jet brains' success in the space alone.

HashiCorp made a $270 million loss last year, more than 50% of their revenue.

A lot of the money comes from Vault which is open source but only used by larger organizations that want support.

Vault namespacing also requires an enterprise license, so yeah if you're doing anything useful with Vault at scale, you're paying for it.

im sure they have a ton of ideas. my bet is that they might launch something in the CI/CD space. like building python packages/images fast, speeding up python testing or tackling something in ML inference.

Telemetry probably. Remember the telemetry in your terminal folks, on here? :cry:

But you might be able to do support contracts, like ansible. Tough road though.

Support contracts. Custom features built to order. Other commercial products based on the open-source libraries for parsing and analysis. Running a hosted service.

All this while remaining a team of 2-3, hopefully, with all company-running stuff minimized and outsourced.

Look at sentry

Sentry requires a platform to work, it's much easier to monetise. I'm not even sure it's possible to self-host?

I think we're going to see a shift in developers expectations towards the cost of their tools, as a side-effect of Copilot & co. If you already pay $100/mo in subscriptions for IDE, its plugins, and services it integrates with, then forking additional $5 for a linter doesn't look as absurd as it used to...

if it's any consolation, Accel is in my mind one of the "good ones". they have a record of backing "good" devtool companies that manage open source and commercial concerns well - Vercel, Sentry, etc. (i'm sure theres more, those are just the two that i'm closest to)

What are the good VC companies? Not just for dev tools, but in general. Where do you find this information?

just being "in the scene" for a few years :)

tbh i care more about individual investors than the fund brand, but its a fallback whenever meeting someone new

edit: https://overcast.fm/+t_0aYbn-o/12:00 listen to steve krouse talk about how accel encouraged him for val.town even tho he didnt initially believe in it himself. thats not normal investor behavior. look out for people like that, by definition they are rare

I actually cursed Maturin a few years ago, as it refused to compile on OpenBSD and hence broke some lib I planned to use.

Python is becoming a bit too reliant on Rust. Rust is good but, in term of portability, is just not as mature as C. If your lib relies on Rust, please please please test it on something beyond Linux and Mac.

> please test it on something beyond Linux and Mac

That’s asking a lot. If your platform is poorly supported by Rust, then maybe that’s where your efforts should be directed. If you fix that, lots of interesting stuff beyond one library is unlocked.

Theo de Raadt engaging the Rust community would be a sight to behold :)

I am not aware of him saying anything about Rust beyond this: https://marc.info/?l=openbsd-misc&m=151233345723889&w=2

Which doesn't mean he hasn't, of course!

That email (from Dec 2017) ends with “Such ecosystems come with incredible costs. For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.”. I presume Theo is actually complaining about using more than 3 GB (?) of memory, but still it really shows the different cost-versus-benefit decisions that we all make.

Yes. My recollection is that around that time, llvm and/or rustc itself would sometimes go over that. I don't know off the top of my head what memory usage looks like right now.

Is it actually reasonable to expect compiler toolchains to work on old or underpowered hardware? Or is the real problem that cross-compilation is still a special case, rather than being the only way compilers work? If it wasn't still normal to depend on your target environment being the same as your build host, would anyone really want to do serious development work directly on their RPi/Gameboy/watch/whatever, just because that is the target runtime environment?

I am wildly guessing that Theo’s beef is more that rust uses a lot of memory (paraphrase: 640kb should be enough for anybody). OpenBSD does integration builds on a variety of different systems, and maybe Theo noticed the OpenBSD/386 build failing due to lack of necessary memory?

That seems a reasonable guess, but it brings me straight back to wondering why such a diversity of build environments is necessary or useful. And my wild guess is that it's mostly because cross compilation is still a second class citizen in most languages today. Though I guess it could be a kind of cultural expectation that you should be able to compile the whole OS on the hardware you're running it on.

My understanding is that the policy of the project is that the base system must not be cross compiled. My understanding of why is that they want to be able to fully bootstrap from base itself.

The rust compiler runs on i86, the complaint is that it can't compile the rust compiler, because it uses more memory than is addressable on i86.

That’s what the person was getting at. Does it matter if it can be cross-compiled elsewhere?

It matters to Theo, due to policies created for specific objectives. It does not matter to many other people.

I guess at the end of the day this is all that really matters. If there are specific goals that Theo has around this, then there's no point in me second-guessing whether those are "reasonable"; it's a matter of values and preferences and whatever.

Whereas if this conversation was about something with broader stewardship, like _Linux_, I'd be saying this is silly, you shouldn't be compromising other things just so you can build your RPi kernel on an RPi.

I think latchin onto this example misses the crux of his raitonale:

> In OpenBSD there is a strict requirement that base builds base. So we cannot replace any base utility, unless the toolchain to build it is in the base.

> Such ecosystems come with incredible costs.

Basically, the cost of adding rust to the OpenBSD base system currently far outweights the proposed benefits (reimplement fileutils), especially considering people will probably not want to pour in the effort needed to rewrite those with strict POSIX compliance (which is another requirement).

He's not saying rust isn't useful but that it wouldn't be a net benefit to have a hard dependency on it in the base system. In the BSD world folks take a very strict and conservative view of what can go in base (and for good reason IMHO).

This is different from the GP comments about just making it possible to use rust programs/libraries in OpenBSD, so we're definitely on a tangent here.

> please please please test it on something beyond Linux and Mac

Why? Simply because you use something that even 95% of the unix folk do not?

I can totally see how they wouldn’t be concerned with that. Just as I’m not concerned with making my JS lib work for those weirdos that still run IE 5.5.

>If your lib relies on Rust, please please please test it on something beyond Linux and Mac.

No, thank you, I'll just add Windows. Your choice to use platforms that nobody else does save for a few specialists does not constitute a need on my part to support your favorite. In the same way that if I package for Debian, Ubuntu and RHEL, it's your problem that it's not on Arch or unsupported by your custom Hannah Montana Linux. Feel free to submit PRs though.

I thought Black was an auto-formatter, similar to clang-format, but for Python. Ruff seems to be only doing linting, not formatting. Am I missing something?

Ruff will be getting more autoformat capabilities: see the link in parent poster and also read the first comment there, from one of Black maintainers.

Pretty bold to try to replace Flake8 and Black with one tool. I hope they succeed. Would be great to have a tool that does both and is substantially faster.

Indeed.

I do hope they'll expose Ruff as a Python module / API in the future. I'm currently using Black to format Python code that's in-memory (never gets written out to disk).

With Black (as an imported Py module), it's just a matter of passing in a string and getting one back. With Ruff as it is now, I'd have to write that out to disk, spawn Ruff process, then read the formatted file back in. Do that for many files and the speed advantages disappear, and actually it's slower.

You can mount the file system in memory for that, but I see your point.

Why doesn't ruff reading/writing from stdin/out, combined with passing the data in with python subprocess.run and os.pipe work here?

    cat bad.py | ruff check --fix --stdin-filename stdin.py - > good.py

Substituting bad.py and good.py for in-memory os.pipe objects for your data. This still involves spawning a subprocess, which can add up, but you're not having to read/write to disk at least.

It does work. The tradeoff is:

* run ruff command many times (with different input file each time) * write all the files to "disk" (tempfs actually so it doesn't involve disk access) and run ruff once

Currently with black I avoid that by importing it as a module once so I get the benefits of both (and is simple to boot).

In other comments here I've read that a person heavily involved in Python-Rust interop is also part of the team, so there's hope they'll expose ruff as a module too and magically whisk my dilemma away :-)

    ruff check . —-fix

Ruff is great, but it lacks a lot of the rules[1]. As far as I understand it reads every file in parallel, if that is still the case there are categories of issues it cannot detect. In my case ruff cannot fully replace my existing tools, but makes for a great companion.

1: https://github.com/charliermarsh/ruff/issues/970

just starting to dabble in Rust, and I've seen PyO3 mentioned a number of times for writing rust bindings. What's the difference between Maturin and PyO3?

PyO3 is the library that enables Python <-> Rust bindings, Maturin is a build tool for packaging PyO3 Rust libraries (which export Python APIs) as Python packages!

thanks!