Fluvio: A programmable data platform | Hacker News Reader

Fluvio: A programmable data platform

infinyon.com

59 pointslukastyrychtr15 comments

> Most stream processing frameworks available today - Kafka, Pulsar, Flink, Spark, etc. - were born in the Big Data era (...)

> These Java-based stream processing platforms

Maybe you mean JVM based? That's not the same thing. E.g. Kafka is written in Scala, not Java.

> Java-based systems demand significant CPU and memory resources, making them unsuitable for extending stream processing to edge devices.

Well okay, but there is a big difference between doing stream processing on a small device and in a central data platform. That comparison doesn't make so much sense.

> Jars were designed at the dawn of the Internet when Browsers were rudimentary HTML readers and programs required runtime applets (aka. sandboxes) to operate. These sandboxes have been riddled with security vulnerabilities

Wait - look at Android. It runs applications on the JVM (well, their own, that is) very successfully. I think it's the opposite: the JVM comes with a lot of power regarding the control of execution. This is a pro, not a con in this context. You will have to rebuild some of that if you use Rust.

In addition, while JVM applications might be quite memory hungry, the performance isn't very far from native code most of the time, thanks to the optimizations that happen at runtime.

Don't get me wrong. Rust is a great language and I'm sure it can gain you significant performance improvements when enough time&resources are put into it. But this post seems to have an overly negative view on the JVM and its ecosystem, which makes me wonder if the motivation for building Fluvio is really in the right place.

Agree. The JVM is annoying to start up because it takes forever but once it's running, Java code has (in my experience) been very fast.

I was having a conversation a while ago about how the JVM is best in breed because at one point or another pretty much every luminary in the field of VM design including JITs and whatnot worked on the JVM.

> The JVM is annoying to start up because it takes forever

In the wrong hands, yes. I didn't touch java for 5 years, but the start time was under 1 second. Is that forever? Look now at Quarkus and GraalVM.

Main difference I can see between all these JVM based frameworks and whatever’s being announced here is that they exist, so maybe wind your neck in a bit.

If we are talking about "cloud-native", I would really recommend taking a look at nsq or nats. I am really big fanboy of NSQ [1], it is simple, easy to setup and conceptually easy to learn.

[1] https://nsq.io

Found the GitHub page … https://github.com/infinyon/fluvio

Looks interesting. Yet the getting started link is broken and seems still quite alpha.

Looks like "Get Started" was broken into Mac/Linux/Cloud

https://www.fluvio.io/docs/get-started/mac/

https://www.fluvio.io/docs/get-started/linux/

https://www.fluvio.io/docs/get-started/cloud/

Hi I am author of the blog as well as contributor. Sorry about links. It should be fixed. Try again

An implicit sentiment of the post is how hard "big data jvm" systems are to learn and work with, as an individual developer.

The open source tools of old were all "apt install" and begin. These "ageing modern" big data systems were hacked together on the massive data centres of internet giants and very poorly packaged for use anywhere else. This has created high barriers to entry for would-be junior data engineers.

I support any system, eg., this one, which brings back this ability to scale naturally up from a one-dev-machine prototype.

Thanks. A key reason for starting Fluvio is to democratize data. Make data streaming & data engineering accessible to everyone and remove barriers much as possible.

What would be good is to have the value prop in a 3,4 sentences up top.

I've gone through 3/4 the post but still I'm not sure what it does concretely.

Also back up your claims that X is better than Y.

It's a streaming-based data platform which means "data stream" is a fundamental first-class citizen similar to how the table is on the relational database. The key benefit is that data can be move without doing polling (or writing job scheduler) with is how done in the traditional batch-based ETL. Compared with the legacy streaming platforms, Fluvio fits better in a cloud-native environment, with less resource required in terms of memory and CPU load usages which means it is significantly cheaper. With our WASM inline programmable engine, you can perform data computation near storage, eliminate unnecessary data movement, reduce network transmission costs and delays, and eliminate security risks. We will start to publish our performance and other stats. Our SPU (Streaming Processing Unit) runs in 1/10 of memory as a Kafka broker (which requires a min of 1G memory) on a simple benchmark.

I have read the article and I do have some questions and observations.

> Some admin operations such as setting up data sharing or re-balancing stream after config update require an IT ticket to be handled by the operation team.

No, they don't. The organizations may have such requirements but the products do not require this. I don't see how this would be different in an org that decided to deploy the Fluvio product.

> Java-derived languages remains the only reliable way to customize stream processing.

There is a reason for that. Java has by far the most convenient method of providing and replacing dependencies at runtime while also providing type safety and being OS independent. Go modules do not count because they're compiled into the program, go plugins - do they actually work? - have serious drawbacks. Things like goplugin from HashiCorp are really nice but they run separate process and communicate over a socket, thus add latency. I don't think it's different in the Rust world. With Java, I can have a program deployed and simply replace a jar dependency, there's not even a need to restart a program in certain cases thanks to class loading mechanisms.

> One of the most significant drawbacks of Java-based stream processing frameworks is the Jar wrapper required for distribution. Jars were designed at the dawn of the Internet when Browsers were rudimentary HTML readers and programs required runtime applets (aka. sandboxes) to operate. These sandboxes have been riddled with security vulnerabilities, and new browsers are gradually deprecating them.

What does the browser security have to do with jar being the distribution model for Java programs?

> Some frameworks resorted to container technologies such as Docker to add another layer of isolation and a workaround for dynamic loading. Unfortunately, the container introduces another layer of security issues and introduces more latencies and cold-startup time.

Sure, Fluvio says it's cloud native and designed to run in Kubernetes. I guess it has the same problems?

> The data-at-motion stream processing must handle an order of magnitude higher data than the products storing data-at-rest.

Are you sure about that? The storage layers I have seen have significantly more data than what's in flight.

> Consequently, stream process platforms must be small enough to boot within milliseconds and operate efficiently on any system architecture. Moreover, it must support a variety of deployments from small organizations to large enterprises. The ideal platform has: ...

What would be the need for booting within milliseconds? What benefit does it have considering that a data broker lifetime is long enough to justify a longer boot time? Do you take consistency into consideration? How is Fluvio handling replication? What is considered "low latency" and why would existing systems not be able to provide that?

> Stream processing engines with access to data must have a robust sandboxing environment that can enforce access control and protect data records from impacting each other.

Aha, so this has nothing to do with Kafka or Pulsar. This has to do with Spark, Flink, Kafka Streams, Puslar Functions.

> While SQL-based tools may be adequate for querying data-at-rest or a data lake, they offer limited functionality for developers who need full API access for automation.

All existing frameworks have an API.

> Development APIs must be available in many widely used programming languages such as Node/JavaScript, Python, Go, Ruby, etc.

I'd be happy to have a fully supported client. That would be enough.

> ... consumer and producer as with legacy Java-based stream processing frameworks ...

That's some "modern vehicle manufacturer" level cockery right there.

> The Rust language powers Fluvio. Rust, a modern programming language built for speed, low overhead, cross-platform interoperability, and code safety. AWS, Mozilla, Google, Facebook, Discord, Dropbox, and others use Rust to create a new class of high-performance products, such as browsers, chat servers, network proxies, database servers, real-time systems, and more.

Whatever. Really, it does not matter to me what the product is written in. I'll be interfacing with it via the client and as long as the platform works, I do not care what is it written in. A programming language the product is written in is not a selling point.

> Rust is safe by default programming language, unlike ...

Again, why would I care when I am interfacing with the product via the client? Those are the reason why the company behind this product have selected Rust and that's great. But it's not a selling point.

> With a Java-based streaming framework, it is challenging to provide a programmable pipeline with performance and security.

Why?

> It is proven W3 technology to bring programmability to software such as Envoy Proxy, Cloudflare worker,

While WASM is great, it's not a standard in those systems. WASM is one of the plugins in Envoy and one of the Workers runtime. WASM is cool but the tooling is not there yet to take it to the real masses.

> Since user modules can only access data supplied by SmartStream, it can’t access or modify protected information as PII data.

How does it prevent that if a stream being processed contains PII data?

> WebAssembly supports any language with bindings to the LLVM toolchain - Rust, JavaScript, Python, Ruby, and Go. Fluvio offers abstractions, templates, utilities, and tools to make it easy and convenient to build and customize stream processing modules.

So basically, I need to buy into the ecosystem and I'm locked in into the WebAssembly stuff used by this platform to achieve the Zen of "performance". Kafka clusters I work with ingest 400MB/s easily and give me an end to end latency of 100ms with additional processing in the way. What is WebAssembly giving me here?

Now, I like the idea of having a lightweight stream processing engine where each topic could be its own raft group. Something that could be deployed close to the storage to avoid having to move volumes of data for processing across network boundaries just to do the computation. However, this article does not suggest that Fluvio would be just that. It mixes the stream processing part with storage part. Kafka is not "stream processing", Pulsar is not "stream processing". I do not really understand what does this do. It seems to be doing everything...

I'm not trying to say the product is bad, but this article does not give me any incentive to look at it. The amount of responsibilities packed in this write up make it sound "complex" and "difficult". It completely glosses over the data storage concerns, replication and high availability of stream processing. I'll keep an eye, though :)

Today, my first go to for a data mesh transport would be AMQP 1.0 because every participant can be both: a client and a server.

There is a significant difference between doing stream processing at edges (client, producers) vs. inline. Similar to performing SQL query in client vs. done by the database server. Think of the WASM inline module as a stored procedure but for the stream (omitting other implementation details).

> How does it prevent that if a stream being processed contains > PII data?

At the execution time, the WASM module can't access any memory outside it is given. Fluvio's SPU control data flow from the stream to the WASM module. For the PII use case, data is only decrypted just before sent over to the WASM engine. And since the WASM module can't access any data or have access to any host-based resources, it can't leak anything.

The key architecture advantage of in-line processing is that we can move computation as close to storage as possible reducing cost and latency

Does InfinyOn Fluvio integrate with AWS InfiniDash?

Reply on news.ycombinator.com