Hacker News Reader

Comment by bitsondatadev | Hacker News Reader

Why is this just near real-time? If we're talking about Flink + Kafka, isn't that considered really real-time?

Maybe near real-time is just considering ingestion and query latency?

This is more of a definitional thing. The default parameters provided in the blog example provide roughly a 60s time-to-query latency. For most folks, this is probably effectively real-time, but for others it might be too slow for other use cases. That being said, this time corresponds to the underlying Flink flush interval and this value can easily be tuned to whatever value you want it to be. For example, you could make it 10s if you wanted. However, if you do reduce the flush latency, you may want to increase your compaction frequency to balance that out.

I’d personally have a hard time calling real-time something that doesn’t react to changes, ie if you need to re-run the query in trino on a schedule to get updates then it’s necessarily near real time (as near as your schedule interval allows). But other people may use it to mean something else

The article talks about the reverse direction however. i.e. Data from Kafka being ingested via Flink for Trino to query as Iceberg tables - everything with a single consistent schema.

To be pedantic though Flink also operates on a flush interval so not "real time" by that definition.

Ah this makes sense. This is similar to what Elasticsearch shards do.

Reply on news.ycombinator.com