We built our data transformation engine with the Wasm runtime | Hacker News Reader

We built our data transformation engine with the Wasm runtime

redpanda.com

34 pointssorenbs21 comments

This is a good post but besides the non-organic upvotes, it looks like your friends/fans/teammates have seeded the thread with comments and questions. This is a super bad idea. HN readers can smell this kind of thing in the parts per million and will flame you for it. It's also against the rules—this is in both the guidelines and the FAQ:

https://news.ycombinator.com/newsguidelines.html

https://news.ycombinator.com/newsfaq.html

i got a notification in slack. there is less organization than you think :D. if we were, we'd have done this w/ the last 15 posts.

Slack messages are how most of this works. It doesn't change the point.

dunno what to say. the answer doesn't change.

Nice post!

Question for you -- what are the performance implications of "re-keying" my records during the transform? Should I try to keep the same key for my transformed record as from my original record so that they align with the same partitions, or are they likely going to get sent to different partitions across different brokers anyway?

Good question! You can't re-key your records but this done is by design. We want the materialized partitions to reside on the same brokers as their sources so there is no network overhead for performing the transform, and it occurs on all nodes. So by choosing this design we traded off more CPU for lower latency and less network bandwidth.

We have been running hundreds of services that effectively consume from various topics, perform mostly simple transformations and produce back to other topics. Providing this kind of functionality makes a lot of sense.

We are using TANK, and that functionality is not available there, but given the benefits such a feature provides, we will likely use RP as well in order to save on development time and number of distinct services we use for various events streams transformations.

Hi I'm one of the authors of the blog post, happy to answer any questions here !

Hiya! Excited to try this, The devil is of-course in the details :) What are the throughput implications for the rest of the cluster? IS there any way to profile the transforms usage as a percentage of the whole? How much of the API is exposed? can it create topics?

Hi, of course there are :) So the transformations are deployed in a separate NodeJs process per node. You can use familiar tools to take a look at its cpu + memory usage.

As for how it affects redpanda, currently we are working on exposing metrics via Prometheus. All of this work occurs on background threads, so the only way it would affect throughput is if enough coprocessors are deployed with enough actual work that would eventually start to fill up our seastar reactors.

can materialized topics have custom properties? what if I want a longer retention in a materialized topic, perhaps I'm just counting clicks and I don't want the whole header data from that topic.

Currently they cannot, as they share all of the same properties as their respective source topics, but this idea is good! I'm adding it to our list of design enhancements.

That's awesome! I would love to try that option one day! Thanks!

Are there other wasm runtimes to consider? Why v8?

Yup, as of now we are leveraging Nodejs for the technical preview. We also have another preview using embedded v8 for inline transforms, that is your transform runs in-process and apply() boils down to a function call. We are still evaluating other runtimes like wasmer as well, v8 was initially chosen for its maturity and community support.

Great minds think alike! We're adding Wasm support to ScyllaDB, too.

https://github.com/scylladb/scylla/pull/9108

Folks can hear a talk on the implementation at Scylla Summit next week [free to register and yes, will be available free on demand after].

We'll also have a talk by Redpanda's Denis Rystsov.

https://www.scylladb.com/summit/

I'm pretty excited to see the wasm chat too. very cool. I think that wasm is going to do to server side what javascript did for the web in the 90's

Reply on news.ycombinator.com