Show HN: Fluvio – Distributed stream processing system written in Rust and WASM | Hacker News Reader

Show HN: Fluvio – Distributed stream processing system written in Rust and WASM

github.com

58 pointsdebadyutirc9 comments

How does Fluvio compare with Arroyo and Timely/Differential Datadlow, or, of course, Apache Flink?

Fluvio Stateful Service Development Kit is in Dev Preview. State operations, windows are part of SSDK.

A decent analog for Fluvio the core functionality of Kafka + Flink + Airflow. Distributed streaming + stateful stream processing + workflow orchestration.

Arroyo and Flink are great stream processing engines with SQL interfaces. Flink is more mature and Arroyo is fast and light since it is written in Rust.

The 2 key differences are: - The implementation of Fluvio SSDK is using the wasm component model, which natively enables polyglot interfaces. Fluvio operates using YAML to express schema, operators, state, and windows. - Fluvio SSDK provides a scaffolding for composing end to end event streaming flows in a single composable paradigm to reduce context switching and make the process of development simpler and uniform.

Without taking the merits out of the effort, for something in previous bytecode formats.

https://www.usenix.org/conference/fourth-annual-usenix-tcltk...

https://en.m.wikipedia.org/wiki/Jini

Thank you for the comment. This is certainly not a novel concept. Its an opinionated implementation that is geared towards solving some business problems.

I'm a bit confused why wasm is involved. What's the application of it in fluvio that makes it the right solution?

Stream operations can be defined ostensibly in any lang. that compiles to wasm, enabling polyglot stream processing.

Yes. This is on the money.

Wasm component model enables the stream processing operations to be natively available to wasm compatible languages.

Fluvio core streaming engine is implemented in Rust.

The first generation stream processing, transformations were initially implemented in Rust needed binding generators to work in Python, JavaScript, Go etc. Also state and window operations and offset management needed to be expressed in application logic.

In the current generation of stream processing wasm component model enables a native support for polyglot stream processing.

Hm, docs only mention node, can't it be used in Deno using the WASM?

Deno can use the fluvio crate directly https://crates.io/crates/fluvio.

Reply on news.ycombinator.com