Hacker News Reader

Comment by remram | Hacker News Reader

This is great but you should be aware of the cost. Distributing binary wheels for every available platform (including ARM, MacOS, ...) and implementation (PyPy...) is not easy, and not doing it causes really abysmal user experience (doing `pip install requests` and being told you need to install a Rust toolchain to build `cryptography`). Sometimes the performance might not be worth it.

Thankfully there are GitHub Actions and similar tools that help with this.

View on HN

I would however make the argument that if you do have a native extension but you have no binary wheel, it's better for that code to be written in Rust than C. Usually for a rust binary extension to build you just need a recent rust compiler on your machine whereas making C/C++ things to work can be a nightmare from you needing to have the right version of cmake, Cons, header files, dependency libraries and more to be installed. I have a text file with common CFLAGS and LDFLAGS I need to set to install various database drivers and more.

I tried once building a very simple Rust extension on Windows using the Anaconda Python distribution and had no luck. If I remember correctly it got stuck trying to find the VC++ compiler.

Cython extensions on the other hand work fine with the MinGW GCC compiler provided by Anaconda, although it's pretty old (version 5).

In my experience that is just the silly amount of variables you have to export to use the VC++ compiler and this is why they provide a dedicated terminal prompt for the VC++ compiler with all the right env vars loaded. I've struggled as well automating VC++ compilation in rust as a result.

If you're so inclined to revisit this using mingw with rust you can see how here https://rust-lang.github.io/rustup/installation/windows.html

This is far easier to automate imo

Finding where & how to use an installed VS instance (or selecting one) in automated tooling is solved by the criminally unknown, MIT licensed, MS supported, redistributable, vswhere tool: https://github.com/microsoft/vswhere

There used to be a "vcvarsall.bat" script that set everything up properly without opening a separate shell session. However I recall that it moved locations on every VC++ update. Does that even exist anymore?

I forgot to mention that I didn't have VC++ installed at all (because it's not free from all points of view).

If I remember correctly I tried installing the MinGW based Rust toolchain too, but I'll try again with the instructions you've given me.

Generally for scientific libraries where this is common, the expectation is from many users that you'll make it available via Conda Forge/Spack/EasyBuild for easier distribution, and that anyone wanting anything else will either get a manylinux wheel targeting a lowest common denominator machine, or will have to build from source.

Having to install a Rust toolchain is still easier than trying to get C extensions to build. Before binary wheels were common on PyPi, I would either give up as soon as I encountered a C extension or switch to something like Conda to get their binaries.

Maturin wasn't powerful enough for our use cases, so we used raw setuptools-rust and cibuildwheel. Even with abi3 compat, yes it's not always easy to get the wheels exactly as you want.

And with a lack of ARM runners by default with GH actions, you'll most likely be paying for your own CI instances (or wait forever for cibuildwheel cross compilation/qemu). Also for others doing this, the rust-cache GH action saves a lot of rebuild time too.

Yeah, Python's dependency management is already abysmal, but when lower level language modules get involved its suddenly whole different level of hell. This actually put me off Python entirely. Sadly no real alternative in ML.

abysmal compared to?

Rust (what I consider to be gold standard at this point), Java, C#, Clojure (piggy-backs on Java/Maven).. Hell, even Emacs-lisp has better dependency management because lack of dependency version control has established strong backwards compatibility culture within community.

Python has neither standard tooling (de facto nor de jure), has no consensus within community where everyone seems to invent their own yet-another-dependency-management-tool and no culture of preserving strong backwards compatibility (never mind the core 2->3 transition).

I've never needed to use anything beyond pip in my 9 years of programming (science, ML, back-end web development). But maybe things get bad when you move beyond those things.

But we'll agree python's dep management is leagues ahead of javascript and R, though. Right?

Yes, this. If you run something that's not in the binary cache you are an in a lot of pain because of Cargo. I have no idea how it works but it's painfully slow (is there some sort of network speed limit implemented? or is it downloading entire git history? )

I have patched too many python programs to remove unsupported cryptography imports. That rust recompile/install is not fun.

Reply on news.ycombinator.com