Write Python inline in your Rust code | Hacker News Reader

Write Python inline in your Rust code

docs.rs

174 pointsAissen39 comments

I'm in the process of porting some DSP code from python to rust and was really wishing I could use matplotlib to check my intermediate data. If this works like it looks like it should it would totally blow my mind!

Matplotlib was one of the reasons I made this. Works just fine: https://github.com/dronesforwork/inline-python/blob/aebabc88...

I liked rust already, but this is amazing!

this just went from cool toy to really useful

Neat example of the power of macros, but actually using it in a project you want to maintain seems incredibly code-smelly.

Author here. Let me know if you have any questions. :)

How does this interact with Rust memory safety? I am not familiar with the PyToObject trait, so I am guessing that this has some kind of safety wrapper?

In Python, all data (even simple integers) are all allocated on the heap. ToPyObject makes a deep copy of everything, converting everything to `PyObject` on the heap, so all data is owned.

The numpy crate helps if you need to convert large arrays to Python, as numpy does not store every element separately on the heap.

> In Python, all data (even simple integers) are all allocated on the heap.

Total sidebar, this leads to one of my favourite programming language "quirks" - you can redefine the value of 1! http://hforsten.com/redefining-the-number-2-in-python.html

Does every invocation create a new Python instance, and global namespace, etc?

It uses the same Python interpreter, but each macro invocation is considered its own module. A later version of the crate will have the possibility to keep the context around to be re-used by a later invocation such that, for example, you don't need to import things again.

So this doesn't actually compile any Python, it just launches a Python interpreter when it encounters Python code?

If I have many threads running Python code will it launch a new interpreter for each thread?

Does it use Python 2 or 3?

Thanks! I might actually use this sometime for scrap code that I'm writing to test stuff.

It doesn't translate Python to Rust or anything like that, it uses CPython both to compile to Python bytecode and to run it. It doesn't launch a separate interpreter, the interpreter is used as a library, so runs in the same process.

Python doesn't really do multithreading: https://wiki.python.org/moin/GlobalInterpreterLock

It uses Python 3 of course. Using Python 2 these days would be a crime.

Cool hack.

Though I'd definitely advise wanting to do this to Do The Right Thing™ and just put each thing in its own file.

I imagine the common usage would be to write an ordinary python module and then just have a small block of inline python in the Rust code that imports and calls the module.

Yes, I think adding a programming language like this is a bit tricky.

But using it to embed markup is really cool.

https://github.com/bodil/typed-html

I remember a while back I saw something where you can do javascript inline with your python. You know with javascript you can do jsx and inline html with your javascript.

This is the ultimate combo making for the ultimate language. Inline html, inside javascript inside python inside rust. This is the future of programming.

There are also crates to use JavaScript inline in Rust. Rust also has fairly complete bindings to the DOM APIs that can be used when targetting web assembly :)

Wow. Next step is to inline the entire JVM and all languages related to it like clojure and scala into rust. Then take rust and inline it into golang then take golang and inline it back into rust.

I can't wait for the future.

Cool, but can anyone explain to me why I'd want this?

I guess it could be pretty cool to make use of a python library from a Rust project.

You're sacrificing the speed and low level nature of Rust though. If performance is not critical why would one choose Rust in the first place? It's neat, but I don't see any need or use case for this.

At work, we make high performant drone control systems in Rust. But for testing and debugging, we simply use Matplotlib (a Python library) to make plots of simulations, etc. So we prefer Rust for the control systems themselves for the speed and reliability, but for plotting results of simulations, we don't care much about performance and rather just use something with all the features we need.

My grandpa used to make drone control systems with operation amplifiers (in the 1980s). I have a picture of him with a big plotter on a stand in a field flying a drone.

May I ask where you work?

Seems like: https://github.com/dronesforwork

90% of time is spent in 10% of code(for some value of 90/10). As a concept, this will give you freedom to use extensive Python ecosystem, flexibility to write glue code etc. in a more 'accessible' language.

Then I guess you're not looking hard enough. Recently swift integration with tensorflow also introduced easy way to import python. There into swift. Data science engineers need performant language to do lot of stuff, and rust fits the ball. But it's hard to abandon python ecosystem.

I and others hate the pattern in python to write anything performant in C, Cython or some other language and write bindings for python. The python as binding language paradigmn doesn't work when all the stuff you want don't exist in performant languages.

TLDR; this fits the perfect use case for machine learning/data science, to use python only when necessary and hopefully migrate projects (parts of them atleast) to rust.

To gain access to thousands of Python modules would be my guess. A billion years ago, I wrote PyInline do that you could easily write C code in your Python program. I was stunned at how many people actually used it for real projects.

I don't know if it's possible, but the first thing that comes to mind would be testing a Python extension written in Rust without having to leave the Rust testing ecosystem.

The author mentioned on Twitter that it was useful for quickly plotting data in Rust, with this example:

    fn main() {
        let data = vec![(4, 3), (2, 8), (3, 1), (4, 0)];
        python! {
            import matplotlib.pyplot as plt
            plt.plot('data)
            plt.show()
        }
    }

Source: https://twitter.com/m_ou_se/status/1118255841961025539

This snippet right here might actually convince me to learn Rust. I work in C++ a lot, and deal with problems where visualization would be useful if it were easy. But because nothing in C++ is ever easy, I end up not doing it in a lot of cases, sometimes to my own detriment.

Sometime, it's just easier to get something working in Python. Plug it into your rust project, see how this holds, and if you are satisfy, convert to rust (or not, maybe it's enough).

Because you want to program in Rust but have a specific need no Rust library full fills but Python libraries do. Suppose you need to convert between Gregorian, Julian, Islamic, Hebrew, and Armenian calendar dates. No such library exist for Rust but they do for Python. So you either spend (waste) time writing one for Rust or you use this Python binding.

This could give a nice way to build desktop apps with rust - make your GUI in Python but the guts are rust. Possibly more straightforward than building libraries in rust and generating python bindings.

I've been looking into doing this in Go and haven't found a good solution for Python 3.

Is it possible to build a static binary (including Python) with pyo3 in rust?

It doesn't seem like it, given the final bullet point here, which is unchecked https://github.com/PyO3/pyo3/issues/276

Doesn't Go have a C FFI? You just call libpython. I'd imagine adding syntactic sugar is harder because Go doesn't have the same metaprogramming capabilities, otherwise it should work the same.

I’m not worried about the interop. I want a not too hacky way tool to statically link it in.

Reply on news.ycombinator.com