The code level integration (pybind11) was nice and easy, the issues came later. We found operational problems (multithreading), performance (in particular initialization) and worst: memory leaks and crashes on python VM teardown. Another problem was that python has it's own particular ideas about package management than generally don't rhyme with packages from the OS vendor.
I'm not sure what the angle is in this project, but I don't see a happy ending for projects that rely on both pip and cargo when it comes to maintenance and long-term support except for source-level integration. That is, unless there is some well designed and well thought out strategy to combine Rust and Python worlds that doesn't conflict with the RPM or apt view of doing things.
I've spent tons of time tracking down python crashes from C extensions -- not how a company wants their devs spending their time -- and 99% of the time its just getting the reference counts right so C and python are on the same page. Dependency management of C pointers gets tricky sometimes...
My usual workflow when I'm wrapping a C/C++ library (which I do for 'fun' quite often) is to generate a skeleton of the python module with pybindgen and then hand-tune it until it works. I could write a bunch of custom wrapper code (which I do use when it makes things easier but it gets thrown away after the skeleton is generated so doesn't have to be very robust) and just use the output from pybindgen but that takes a lot more work unless the library falls into pybindgen's happy path. Plus, as I'm doing it for fun, I don't mind spending time to prettify the generated code and add some py-sugar. And the way pybindgen generates docstrings isn't the best so those would have to get handwritten either way.
Admittedly, I just do this kind of stuff as a hobby and industry has different goals so "pinch of salt" and all that.
But I think, the point GP is making is more about the dependencies Python ads to your C++ (or Rust). Ie.: build RustPython project to produce artifact that works on my machine, but relies on my python interpreter, my (which may have C dependencies) as well as a rust edition and cargo. How can I bundle this into a package for a different Linux? Preferably do it following the existing packages and lore of that distro, rather some python and some rustup version?
Exactly! I want customers to be able to install my software on stock Ubuntu 24 / Rocky 9 / .. and seamlessly integrate with existing stock Apache, libcurl, libssl without any recompilation etc.
Having a bunch of pip dependencies tag along makes this quite complicated.
Lua being designed specifically for embedding makes it a good choice.
I wonder, though whether we might have ended up with similar fights maintaining LuaRocks dependencies in rpm/apt ecosystem on various platforms as soon as those customizations start pulling in dependencies like XML parsers, crypto, etc.
Do you have thoughts on that? (Note, we target linux, but sell proprietary closed source binaries only, except for the customizations)
A baking spatula makes a bad cooking spatula. It's pliable and can easily scrape the curved surface of a bowl, but for flipping a fried egg it is hopeless - in a pinch it will work - but it would be nice to have something rigid.
Further, describing it as making you "wince" is a recipe for a flame war bordering on deliberate bating. Just describe your issues with the language in more than a single charged word. You don't have to write an essay, just "I don't like the package management environment". That way people can have a constructive discussion.
Are you trying to say that all tools are the same? There are no bad tools?
Why is it naive for a painful tool to make you wince?
Rust is great if you like puzzles and want to spend your precious time solving the same memory management riddle over and over again. You do get faster and often more robust code though, although in practice the difference is often not meaningful and the extra time invested doesn't pay off.
So, you're left with a choice of C++, Rust (what else, C#, possibly Zig and a few other).
Rust stops being a "puzzle" once you've written enough of it and you just know how to do things. It has one major disadvantage though - returning back to write in languages like C++/Python makes you cringe because they don't have basic language tools like proper sum types or traits that get things done.
The Rust memory management puzzles are not really complicated. They just get in the way.
> they don't have basic language tools like proper sum types or traits that get things done
Sum types and traits are not getting things done.
99% of whose time? Where does this estimate come from?
Definitely not my case, and definitely not the case for lots of programmers I know and/or work with. Python has its own place (for example, in DS/ML) and while I use it myself on daily basis for what's it's best at, statements like "you should be programming just about 99% of your time in Python" make no sense at best.
And, note, it's not only about speed/performance; after writing lots of Rust (or even F#), it's hard to get back to Python because of missing expressiveness and basic language tooling.
Finally, speaking of Python (CPython in particular), part of why it's so attractive is all the fast binary extensions libraries written for it... in C++/Rust, not Python.
> Sum types and traits are not getting things done.
Disagreed, sum types are exactly what gets things done, regardless of language performance and other features. This kind of logic exists everywhere. That's why languages with proper (or even semi-proper) sum types like Rust, F# are a lot easier to express 'business logic' in. And e.g. that's exactly why serialization/deserialization can get so weird in Python where values can be of multiple types and you have to invent crutches to work around that, to simulate sum types in this way or another.
No, not me, because I’m less productive in Python than in Rust, assuming the same level of final product quality, even excluding runtime performance.
> Turns out 99% of the time you don't need to parse terabytes or have μs-response times.
Performance is not the primary reason to use Rust. It has way more to offer than performance.
So much this. I've been doing mostly Scala (and some Haskell) for the last 10 years, and now I'm really enjoying Rust precisely because of its language features. Performance is just a (really) nice bonus.
Other bonuses include simple deployments (single binaries versus massive JVM and fat jars), low memory footprint, simple builds that just work (Cargo is an absolute joy), great concurrency, great ecosystem, etc. etc.
Re: your point, I'd still recommend learning Rust even if it's just solving last year's AoC, to see how various data structures, enums, traits, iterators etc work. Even if you don't end up using Rust on daily basis, it will probably affect the way you structure non-Rust code and provide you with a few neat design ideas.
Do I:
python? python3? pip install? pip3 install? python pip install? python pip3 install? python3 pip install? python3 pip3 install?
And then everyone mentions "oh just use venv" or "conda" or docker or... It just never ends or it never clearly explains what to do. And all I want to do is run a relatively simple script. Or course I would use a docker image to run a more complex app. But it's frustratingly hard to manage for people who don't use it often.
I'm from a java/Kotlin world where things aren't perfect either. But at least dependencies are clearly defined, and the only thing you need is Gradle that updates itself through a wrapper script if needed and pulls all dependencies. And a JVM/JDK which is also configured/installed quite clearly in a specific location.
The simplest way (in the sense of having the fewest components required) is this, using only built-in tools:
$ python3 -m venv --upgrade-deps my-virtual-environment
$ my-virtual-environment/bin/pip install whatever-third-party-package
$ my-virtual-environment/bin/python3 my-script.py
This creates an isolated virtual environment in "my-virtual-environment". Use the pip and python3 binaries inside that directory to install packages and run scripts. Done. (The "activate" step suggested in other comments is just a convenience for setting your PATH to refer to the virtual environment implicitly, so that's completely optional.)Note that Debian's python3 package leaves out some parts of the standard Python distribution, but you can install the python3-full package for a complete installation.
> py -3 -m venv --upgrade-deps my-virtual-environment
> my-virtual-environment\Scripts\pip install whatever-third-party-package
> my-virtual-environment\Scripts\python my-script.py
This assumes that the Python launcher (py.exe) was installed when installing Python, which it is by default.The story about python on windows is a nightmare for new people for these reasons. And installing packages - while not as bad anymore - is a mixed bag due to fewer precompiled C extensions than on linux.
Paradoxically, microsoft have spent some effort on improving things. This has led to multiple incompatible choices which, effectively, has made the experience worse.
If you do use Microsoft's Python distribution though, are you saying that it provides neither "python3" nor "py"? How do you use it then?
According to the docs[0] the Microsoft Store package should provide "python" and "python3" commands (effectively unsolving the problem solved by the Python launcher, I guess?) so either my original suggestion of "python3" or my subsequent suggestion of "py -3" should work, depending on which distribution was installed.
[0] https://docs.python.org/3.11/using/windows.html#the-microsof...
1. [PATH TO YOUR PYTHON COMMAND IN WHATEVER PYTHON DISTRIBUTION YOU YOURSELF INSTALLED ON YOUR OS OF CHOICE] -m venv --upgrade-deps my-virtual-environment
2. [PATH TO PIP COMMAND INSIDE THE DIRECTORY CREATED BY THE FIRST COMMAND IN THE SYNTAX OF YOUR OS OF CHOICE] install whatever-third-party-package
3. [PATH TO PYTHON COMMAND INSIDE THE DIRECTORY CREATED BY THE FIRST COMMAND IN THE SYNTAX OF YOUR OS OF CHOICE] my-script.py
Rust, Go and Deno are definitely part of a new breed of "developer tooling should just bloody work!" languages. Python is from the era of "eh, you figure it out" mixed with a little of "I'm making this up as I go along".
Java? I seem to recall build systems and tooling being more pain than Fortran. Not sure I believe that one. Certainly not my experience.
https://peps.python.org/pep-0723/ might make life much simpler for single-script applications.
Its still new but works well. I'm transitioning to it from an unholy mess of pyenv, pip installs and other manual hacks.
If you're starting a new python project that is more than just a straightforward script I'd use Rye from the get go.
but python? it between conda, pip, pip3, virtualenv and who knows what else. its hard to make something that I knwo will just work on everything.
python3 -m venv venv
source venv/bin/activate
pip install whatever
echo "source path/to/venv/bin/activate && python3 path/to/app.py" > myscript
mv myscript /usr/local/bin
and just use myscript.
I use this for a lot of little scriptsprintf '%s\n%s\n' "#!path/to/venv/bin/python3" "$(cat path/to/app.py)" > /usr/local/bin/myscript
Path/to/venv/bin/python path/to/app.py?
Neat trick though, gonna adopt it!
You should universally use `python3`. Except on Windows the official distribution is bonkers and has no `python3.exe`. Probably the easiest solution is to install it from the Microsoft store instead which does.
With Gradle, where do you define dependencies? All with version numbers in the `dependencies` block? In a separate file, declaring a hash map of versions and then a hash map of dependencies, and then putting the latter into `ext`, which magically propagates to just about everywhere in the build silently? In a separate file using some plugin that tries to replicate the lockfile functionality baked into most other dependency systems? In a separate TOML file, using the Catalog?
...and that's just about dependencies, I could go on for days about other Gradle features...
It's not any better than Python. You're just used to it. The same complexity is there in both Python and Kotlin, you just learned to cope with one better than the other.
(Gradle with Kotlin DSL and good IDE support is a little better, mostly because Kotlin devs are not as afraid of touching the build as when it's done in Groovy. It's still 2 orders of magnitude more complex than whatever you use for Python - unless you use zc.buildout or something like that.)
WASM would be an option, but overengineered for my case. Ruby (through Artichoke), or Python (through RustPython); but they come with the downside of introducing Ruby or Python. I haven't decided yet, but would prefer Lua for it's simplicity. Or JavaScript (a subset) for how easy it can be limited in scope.
Are there any (example) rust programs out there that have embedded scripting, or runtime plugins or addons in a scripting language?
https://crates.io/crates/duckscript
You can use the above links to browse crate "dependents", that is other crates using these dependencies if you are looking for example usage.
Though Lua probably has lots of good and feature rich examples as well.
Can't speak at all to Ruby or Python integrations.
In the end, I'd suggest it depends on your audience and what existing skills or biases they're likely to bring to the table.
My one warning is that I've used software that integrated JS and it hasn't aged well. From the Adobe products with ExtendScript to other, older and incompatible JS engines before ES5 even, don't get locked into something difficult to update and keep current.
edit: to be clear, I did mention "I would prefer lua for it's simplicity".
I'm exploring all the existing work out there. Lua is certainly an option. From what I can see on the outside, is that lua embedding for rust is immature compared to some other solutions. But maybe that's because the lua-folks are just more honest :)
O lord of the compiler, lend us thine wisdom.