OCaml for Kafka stream processing, analytics and telemetry data

github.com

4 pointstingfirst3 comments

For OCaml users interested in data streaming processing (similar to Flink or Spark), but looking for a faster and more efficient option, check out this OCaml plugin Timeplus Proton. Concise, safe, highly performant and fun

-> Streaming Queries - Process large datasets with constant memory usage

-> Async Inserts - High-throughput data ingestion with automatic batching

-> Compression - LZ4 and ZSTD support for reduced network overhead

-> TLS Security - Secure connections with certificate validation

-> Connection Pooling - Efficient resource management for high-concurrency applications

-> Rich Data Types - Full support for #ClickHouse types including Arrays, Maps, Enums, DateTime64

-> Idiomatic OCaml - Functional API leveraging OCaml's strengths

Interesting. How does the performance look like?

Micheal, the author of the project did some performance test and he might annouce it later, but one of his first test show Async Insert with Rate: 3422488 rows/second, but I beleive after optimization and tuning, it could be much faster.

Reply on news.ycombinator.com