How does this compare to and/or integrate with OTOH Apache Arrow which had "arrow plasma IPC" and is supported by pandas with dtype_backend="pyarrow", lancedb/lancedb, and Serde.rs? https://serde.rs/#data-formats
I think it's worth being a little more clear here - Arrow IPC is _not_ deprecated, and has massive momentum - so much so that it's more or less already become the default IPC format for many libraries.
To me it remains unclear what the benefits of Iceoryx2 over the Arrow ecosystem is, and what the level of interoperability is, and what the tradeoffs of either are relative to eachother. Within a single machine, you can mmap the IPC file. You can use Arrow Flight for inter-node or inter-process communication. You can use Arrow with Ray, which is where Plasma went.
I love anything new in this space though, if/when I have time I'll check this out - would love it if somebody could actually eloborate on the differences though.
For other readers, here's a general takeaway -
- Arrow Plasma was deprecated, and is now no longer even present in the Arrow project. - The maintainers behind the Plasma store in Arrow forked it, into Ray. Plasma is still alive and well in Ray - and it still uses the Arrow IPC format, among other things. In addition to the links from the above poster, read the original blog post on Plasma [0], and the section on Ray on [1].
I use Ray quite a bit. For more lightweight stuff, or for more low-level control, I use Arrow Flight [2] (and [3] for Python examples)
[0] https://ray-project.github.io/2017/08/08/plasma-in-memory-ob... [1] https://arrow.apache.org/powered_by/ [2] https://arrow.apache.org/docs/python/flight.html [3] https://arrow.apache.org/cookbook/py/flight.html
Serde does serialization and serialization with many formats in Rust