Hacker News Reader

Comment by urschrei | Hacker News Reader

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct.

https://github.com/urschrei/simplification

https://github.com/urschrei/lonlat_bng

https://github.com/urschrei/pypolyline

Each of those repos has links to the corresponding Rust “shim” libraries that provide FFIs for dealing with the incoming data, constructing Rust data structures from it, and then transforming it back on the way out.

As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line.

View on HN

Given that it's via pyO3, you could even pass the numpy arrays using https://github.com/PyO3/rust-numpy and get ndarrays at the other side.

Same no copy, slightly more user friendly approach.

Further criticism of the actual approach - even if we didn't do zero copy, there's no preallocation for the vector despite the size being known upfront, and nested vectors are very slow by default.

So you could speed up the entire thing by passing it to ndarray, and then running a single call to sum over the 2D array you'd find at the other end. (https://docs.rs/ndarray/0.15.1/ndarray/struct.ArrayBase.html...)

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line.

Not true!

What you do is that you keep a registry for objects passed from the host vm to the foreign vm in which you register objects thus transferred. And you use a similar mechanism for objects passed from the foreign vm to the host vm. In CPython, you simply increment the refcount to prevent Python from collecting them prematurely.

This is how Java does it (via JNI) and how many gc:ed runtimes interact with other gc:ed runtimes. This is how you do it in Rust too since Rust can't tell how long an object passed to a foreign vm is supposed to last.

> This ... is how many gc:ed runtimes interact with other gc:ed runtimes.

It's not that easy. You'd need to register the object as a GC root as long as it's being managed by the foreign GC, and similarly ensure that objects in the foreign VM are properly "de-registered" when the local GC finalizes them. It's far from trivial, particularly when compared with other memory-management strategies.

Thanks – this is the more informed version of what I wrote.

Yes, that is what the scheme I described is doing. You'd need a similar mechanism in Rust too to prevent it from collecting objects prematurely. I think implementing this is fairly trivial. The difficult part of writing an FFI is handling closures: Java calling Python calling Java calling Python which throws an exception...

This is a solved problem: look at how the Lua FFI does it. Lua both is capable of exporting its own garbage collector to its counterparty and has a plug-in system for GC so that Lua can use the counterparty's GC. That allows you to either use Lua's GC for foreign objects or have the counterparty's GC manage Lua's objects.

It's funny the hostility to JIT that some fans of compiled languages have.

But it breaks down if you have cross-language cycles, because no garbage collector of either language will see the full cycle.

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line.

When I was interfacing D code with a part of Unreal Engine that I think is garbage collected, I actually just took the L and copied everything into buffers on their malloc when handing stuff off to the engine. It wasn't particularly hot code so the memcpys were worth the peace of mind I found, ugly as it was.

To add, it is trivial to guarantee that Rust code is "zero-copy", so if this is something you care about, Rust allows your program to fail to compile if it tries to make a copy.

Godbolt example?

Here you go:

https://play.rust-lang.org/?version=stable&mode=debug&editio...

Because the data structure LotsOfData doesn't implement the Clone trait, attempting to create a copy will fail at compile-time.

OK. I was curious whether it was part of the type system or one of those #[feature] thingies.

https://run.dlang.io/is/jOXfJz is a D equivalent

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line.

Safe interop between two GC'd languages (Haskell and Java) was one motivation of Haskell's new -XLinearTypes extension

https://www.tweag.io/blog/2020-02-06-safe-inline-java/

Generally, the new linear types are exciting but are nascent at the moment. But one extension to them (linear constraint [1]) seems to allow embed the equivalent of Rust's ownership in Hsakell using more primitive features in the type system (to my understanding..)

[1] https://arxiv.org/pdf/2103.06127.pdf

I'm also getting a 30% speedup simply from manipulating &PyList references instead of Vecs (without parallelism).

There is no problem at all interfacing Julia's C API to Rust. It's a shame that C++ and Rust are bad fits together, but this actually strengthens the argument for Julia as the extension language rather than Rust, since Julia interfaces easily to not just C but also the C++ code in which much of the world's best numerical algorithms are written, and has an unrivalled FFI to Python, while Rust's object system is a poor fit for either Python or C++.

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line.

<s>As for the FUD about interfacing JIT code to C/C++, this is a problem Julia was designed from the outset to tackle. Incidentally, Julia built on the excellent experience LuaJIT has had. A challenge: can you name a particular extension that would interface better from Rust than from Julia?</s>

Oops, I misunderstood the criticism you were hinting at but failing to justify, sorry. OK, Julia already does interface to Python and this interface sees widespread use. If your suspicion is right, then where are the horror stories from people who were bitten when deploying code that built on the interface?

> There is no problem at all interfacing Julia's C API to Rust

Nobody claimed there was a problem?

> As for the FUD about interfacing JIT code to C/C++

I don't know what you're talking about. Sorry!

I misread your comment. I still consider what you wrote contained FUD, at least with respect to the last paragraph.

Reply on news.ycombinator.com