tingfirst

Makes sense if you prefer scanning the whole picture first. Whole-to-parts vs. parts-to-whole is a different mindset. I personally like fade-in in some cases—it helps me focus on one layer at a time and build up the full context without getting overwhelmed. That said, efficiency can be an issue, so each viz is situational and easy to overuse :-)

Temporal binding is one of the hardest problems in visualization, all about aggregations across different time windows.

And it gets even harder when the clock never stops and live data keeps flowing.

Client visualization layer: Timeplus Vistral (https://github.com/timeplus-io/vistral)

Server data processing layer: Timeplus Proton (https://github.com/timeplus-io/proton)

Streaming-native from processing to insights in motion

All streaming processors face the same fundamental problem:

Streaming joins require maintaining state for both sides of the join

High-cardinality data (millions of unique keys) means huge state sizes

Traditional approach: Keep everything in memory will make memory exhausted

The high-cardinality join memory problem isn't unique to Timeplus. Apache Flink also uses hybrid hash joins that spill to disk (RocksDB) when memory fills, Materialize shares indexed state across multiple queries (but still requires keeping full datasets in memory), and RisingWave stores state in cloud object storage (S3/GCS) with LRU caching for hot data. What makes Timeplus different is its purpose-built optimization for the Pareto Principle, where a tiny fraction of data generates the vast majority of activity - keeping hot data in memory and cold data on disk for dramatic memory savings.

Redpanda + Timeplus, the perfect pair for data streaming developers. No JVM, ZK ...

Probably the smallest yet most powerful binary for real-time, incremental SQL data processing, end to end!

Consistently we heard about ClickHouse has very limited materialized views that can't handle real-time pipeline fast efficiently enough. would love to see more comments here.

Data sources are usually in Kafka, or other operational databases like Postgres or MySQL

1. Table A : fact events, high-throughput (10k~1M eps), high-cardinality

2. Table B, C, D : couple of dimension tables (fast or slow changing).

The use case is straightforward : join/enrich/lookup everything into one big flattened, analytics-friendly table into ClickHouse.

What’s the best pipeline approach to achieve this in real-time and efficiently?

Is there a native SQL pipeline tool for ClickHouse that processes real-time data incrementally, with low latency, large throughput and high efficiency, similar to Snowflake’s Dynamic Tables?

[1] Dynamic Tables: One of Snowflake’s Fastest-Adopted Features: https://www.snowflake.com/en/blog/reimagine-batch-streaming-...

For parallel programming, what's OS-level difference compared to languages like Python or modern C++?

Domain.spawn (fun _ -> print_endline "I ran in parallel")

Anyway, love the simplicity of this expression!

AI can be hallucination but real-time detection is key

re EPS and CPU utilization, WS still performs better than SSE?

For OCaml users interested in data streaming processing (similar to Flink or Spark), but looking for a faster and more efficient option, check out this OCaml plugin Timeplus Proton. Concise, safe, highly performant and fun

-> Streaming Queries - Process large datasets with constant memory usage

-> Async Inserts - High-throughput data ingestion with automatic batching

-> Compression - LZ4 and ZSTD support for reduced network overhead

-> TLS Security - Secure connections with certificate validation

-> Connection Pooling - Efficient resource management for high-concurrency applications

-> Rich Data Types - Full support for #ClickHouse types including Arrays, Maps, Enums, DateTime64

-> Idiomatic OCaml - Functional API leveraging OCaml's strengths

For OCaml users interested in data streaming processing (similar to Flink or Spark), but looking for a faster and more efficient option, check out this OCaml plugin Timeplus Proton. Concise, safe, highly performant and fun!

[1] https://github.com/mfreeman451/proton-ocaml-driver

Pretty cool to see a C++ R/W Iceberg client without dependency, and even better open-sourced. The pipeline is all about processing and routing, ideally, to open and flexible destination with no lock-in and long-term retention. Writing into Apache Iceberg is becoming critical to give users real control, rather than into specific data warehouses or lakehouses that are hard to move out.

Design principles of Timeplus Proton are simplicity, speed and efficiency.

That's why we love ClickHouse, the fastest and most lightweight approach for real-time analytics. Furthermore, data stream processing should also uphold these same principles, without unnecessary complexity.

With Timeplus Proton - a single-binary, fast and efficient streaming processing engine, ClickHouse users can now natively and effortlessly utilize SQL queries and Materialized Views to enable fast and scalable incremental streaming processing from Kafka or any other data stream source, turbocharging broader real-time streaming analytics use cases such as data stream pipelines, unified online/offline ML features, infrastructure monitoring, or any other latency-sensitive applications.

Proton is a lightweight streaming processing "add-on" for ClickHouse, and we are making these delta parts as standalone as possible. Meanwhile contributing back to the ClickHouse community can also help a lot.

Please check this PR from the proton team: https://github.com/ClickHouse/ClickHouse/pull/54870

Redpanda Proton ClickHouse: A perfect match as a single-binary approach for a lightweight and high-performance data streaming processing and analytics in one compact box!

Great points! SQLite as an analogy is fantastic for its small footprint. Swift download, deployment, and testing significantly boost dev productivity. Moreover, the dependency-free single-binary can efficiently slash deployment and operational costs. In addition to its outstanding performance, Proton users are specifically requesting data streaming processing, routing, analytics, and actions at the edge or in a hybrid environment before sending to their centralized data warehouses. This stands out as unique capabilities compared to existing complicated stacks.

As a streaming SQL engine, Proton is a fast and lightweight alternative to Apache Flink, powered by ClickHouse. It can help developers solve the data streaming challenges from processing, routing to analytics, and send aggregated data to the downstream systems. Proton is also the core engine of Timeplus (http://timeplus.com), which is a cloud-native streaming analytics platform.

If you are looking for a high-performance and lightweight streaming processing engine capable of running everywhere, or gaining streaming insights with historical context, you should try Proton: https://github.com/timeplus-io/proton!