Hacker News Reader

Comment by ptato | Hacker News Reader

What is the use case for embedding Python code in a Rust program?

There could be no reasonable use case for something like this, and yet it would still have artistic value and this would still be an interesting article.

It's a brilliant hack and we are on Hacker News after all.

That's exactly what I thought when I read the title. It's very frustrating how many landing pages, project readmes, blog posts, etc that don't answer the question "why?". I usually need to know that before going further when I come across something that I've never heard of before. If they don't have the "why?" somewhere easy to find I just close the tab.

You know, that might not be unintentional. They might want those who don't see the value to just close the tab and move on. That way only those who see the value keep reading. I imagine it helps avoid having to engage in (often endless) arguments about the validity of the use case.

It's a bit of a guilty pleasure for me to read these kinds of weird things with no practical application. It's just a little refreshing to read about something technically possibly, but frivolous (no offense meant to the OP, of course) just because it can be done.

"Because it can be done" is a very solid answer to "Why?"

It helps me to step outside my own context and view a familiar use case, problem, or solution with an unfamiliar perspective.

I am quite surprised by your message. This is HN and even before or outside of that, for years I have myself dived into totally random projects simply because "why not?".

To try random ideas is part of learning and when you have something to share, just do that. There will always be like minded someone to pick up.

Whenever I see someone ask these "whys", I have the urge to show them this.

https://www.youtube.com/watch?v=Y4hOIgRPlNU

Because it was there.

I’m just throwing out ideas but what if I wanted to take working python code and convert it to rust? Could I use this to start as the baseline and start replacing bits of it and see that it still produces the same output?

There was a recent ATP.fm podcast with Chris Lattner of Swift fame. He talked about the topic of embedding Python code in Swift for TensorFlow and it overlaps with your suggestion.

Swift for Tensorflow has implemented Python interoperability, rather than embedded Python code as in this article. A Swift program can import a Python module and interact with Python objects and functions as if they were native Swift objects (minus strong typing, of course). See https://github.com/tensorflow/swift/blob/master/docs/PythonI...

Rust has had Python interop through the Python C API since around 2015 [1]. It's pretty low hanging fruit for any language that supports the C FFI. Rust-cpython has had simple macros for interop for ages [2] and there's even a library that uses serde macros to encode/decode pickled object [3].

This article is just a fun hack using Rust macros.

[1] https://github.com/dgrunwald/rust-cpython

[2] http://dgrunwald.github.io/rust-cpython/doc/cpython/macro.py...

[3] https://docs.rs/serde-pickle/0.6.0/serde_pickle/

A friend of mine has a similar project called PyOxidizer that's intended for building standalone executables with Python+Rust: https://pyoxidizer.readthedocs.io/en/stable/

His primary use for it is building distributable binaries for Mercurial, which is written primarily in Python.

Plugins for example? Game scripts? Think of use cases for embedding Lua into a C program.

Maybe interfacing w an ml lib like pytorch or something?

For pytorch it would probably be easier to do something based off the pytorch C++ interface.

Sure, I've never even used pytorch. But I'm just saying python has a fair bit of data analysis and ml tools that probably haven't found native homes in rust

I suppose you could, but I believe that there are direct bindings from Rust to Pytorch at https://github.com/LaurentMazare/tch-rs. I haven't used it personally, but I've heard fairly good things about it's correctness and responsive maintainer.

I have written a transformer-based (BERT, XLM-RoBERTa) sequence labeler/lemmatizer/dependency parser in Rust with tch-rs [1] and it is great! Some parts do not feel rusty. E.g. dynamic typing of tensors, you get a Tensor rather than Tensor<f32>, but that's more due to how libtorch itself works. But it's a very straightforward API that exposes most of libtorch. The maintainer is also very nice and responsive.

I was also amazed how much is implemented in libtorch itself as opposed to the Python wrapper, which makes much of the Torch functionality available to other languages.

[1] https://github.com/stickeritis/sticker2 https://github.com/stickeritis/sticker-transformers

Someone might have jumped that hurdle but the point is it circumvents that class of hurdle. What about the next framework which doesn't have bindings? What about leveraging existing Pytorch code that's written in Python?

I only had a quick scan through the article, but could this be used to create an executable for some existing Python code?

What would the point of that be versus wrapping it in a bash script? Obfuscation?

Python is missing a nice simple way to ship native executables. If you have to send people a package to unpack that's an extra step. In theory you can use freeze, but configuring it all is a pain.

I've only used it for small GUIs (a couple of files & dependencies) but `pyinstaller --onefile` is about as easy as it gets.

You can use pip to install the executable source files. Pip is preinstalled on most Libux distributions and MacOS.

If imports worked I could see it being very useful for making graphs. I've heard of people serializing data in other languages and then using Python to plot it.

I can confirm this--Rust's data visualization and plotting libraries are still pretty early stage. You can generally do 2D bar charts and scatter plots pretty well, but once you start jumping into more complex representations or three dimensions, you start running into some serious barriers.

Imports work fine. The main reason I wrote this, was to be able to use the Python matplotlib library in Rust:

https://twitter.com/m_ou_se/status/1120577172438233088

Reply on news.ycombinator.com