ArkFlow – High-performance Rust stream processing engine
github.com
github.com
one lesson I learned from this: any bit of logic that defines a computation should prefer explicit imperative code (eg python) over configuration, because you are likely to eventually implement an imperative language in that configuration language anyway
so either it's very small/mature and you don't have to worry too much, or in the active development case your config files are pretty much the instruction set of some kind of logical foggy vm... and eventually a whole environment of tools etc. will "compile down" to your config files and you get a pain knot to endlessly massage...
The customer is the hard part in all of this, but there is respite if you are patient and careful with the tech.
If you are in a situation where you need to go from one SQL database to another SQL database, the # of additional tools required should be zero. Using a merge statement & recursive CTEs per target table, you can transform any schema into any other. Most or all of the actual business logic can reside in the command text - how we filter & project data into the target system.
If we accept the SQL-to-SQL case has a good general solution, I would then ask if it is possible to refactor all problems such that they wind up with this shape in the middle. All of that nasty systems code could then be focused more on loading and extracting data into and out of this regime where it can be trivially sliced & diced. Once you have something in Postgres or SQL Server, you are at the top of the hill. Everything adapts to you at that point. Talking to another instance of yourself - or something that looks & talks like you - is trivial.
The other advantage with this path is that refactoring SQL scripts is something the customer (B2B) can directly manage in many situations. The entire pipeline can live in a single text file that you throw around an email chain. You don't have to teach them things like python, yaml or source control.
Highly recommend checking RisingWave out if you have real time streaming transformation use cases. It’s open source too.
The benchmark was some high throughput low latency JSON transformations.
Benthos: https://github.com/redpanda-data/benthos
DataFusion: https://github.com/apache/datafusion
A contrast between Arroyo and systems like Benthos and from what I can tell ArkFlow, is that Arroyo is a "stateful" stream processing engine, which means that we can support things like windows, aggregates, and joins, with exactly-once semantics and fault tolerance, at the cost of significant additional complexity[1].
[0] https://www.synnada.ai/ [1] https://www.arroyo.dev/blog/stateful-stream-processing
This logic
> written in rust will have better performance, lower latency, ..., lower memory footprint
is flawed and is cargo-cult programming unless you say what are you objectively comparing it against and how you intend to achieve those goals. Picking the right™ language just for the sake of these goals won't get you too far.
Currently, most of the mainstream stream processing engines are written in Java. Sorry, I may not add qualifiers to make you misunderstandings.
Software does not have silver bullets, so does programming languages, and each has its own strengths. I also like to use go and Java to develop software.
There is absolutely nothing in Rust's semantics preventing you from writing high-performance data processing workloads in it, and in fact it's one of the best languages for that purpose. Beyond that, the usual barrier to entry for working on a product like this written in C++ is incredibly high in part because stability and safety are so critical for these products--which is one of the reasons that in practice they are often written in memory safe languages, where C++ is not even an option. Have you worked on any nontrivial Rust data processing product where "copy/clone imposed semantics" somehow prevented you from getting big performance wins? I'd be very curious to hear about this if so.
I'm curious how you came to this conclusion?
In another words, programming language is usually not at the very focus of daily development, given that there's always much bigger fish to fry in this domain, but if Rust provides such an undisputed benefit to your business model, while keeping the cost and risk of it viable for the business, then it's going to be a no-brainer. Chances are that this is going to be the case is very very low.
So, my advice would rather be use the language whichever you prefer but don't dwell over it - rather put your focus on innovating workload-specific optimizations that are solving real-world issues that are palpable and easily proven/demonstrated. Study the challenges of storage or data processing engines or vectorized query execution algorithms. Depending on the domain problem you're trying to solve, make sure that your language of choice does not step in your way.
not_sure_if_serious.jpg
To be extra clear about it (and to avoid pure snark, that's frowned upon here at HN): that's the kind of software (alongside a lot of general enterprise code) that got rewritten from C++ to Java, not the other way around. The increased safety of Java was absolutely a consideration. Java was the 'Rust' of the mid-to-late 1990s and 2000s, only a whole lot slower and clunkier than the actual Rust of today.
Frenzy around the "safety" IMO is way too overhyped and when you and OP say that "safety" plays a huge role in data processing and database kernel source development, no - it is literally not even a 1% of time that a developer in that domain spends his time on. C and C++ are still used in those domains full on.
> that's the kind of software (alongside a lot of general enterprise code) that got rewritten from C++ to Java, not the other way around
Which C or C++ engines exactly got rewritten to Java? We can start from this list: https://db-engines.com/en/ranking
> Which C or C++ engines exactly got rewritten to Java?
It's difficult to give names precisely because private enterprise development was involved. But essentially every non-trivial Java project starting from the mid-1990s or so, would've been written in C++ if it had been around in the late 1980s or earlier in the 1990s. It's just not very sensible to suppose that "data processing" as a broad area was somehow exempted from this. And if writing segfault-free code in C/C++ could be dismissed as a mere "skill issue" we wouldn't need Rust either. It's a wrong take today and it was just as wrong back then.
(And yes, Java took significant steps forward in safety, including adding a GC - which means no wild pointers or double-free issues - and converting "null pointer" dereferences into a properly managed failure, with backtraces and all that. Just because the "safety" vocabulary wasn't around back then except for programming-theory experts, doesn't imply that people wouldn't care just as much about a guarantee of code being free from the old segfault errors.)
I'm one of the maintainers of tremor, happy to get together and talk about rust event processing if you ever want to :)