Hacker News Reader

Comment by jmngomes | Hacker News Reader

>> No one has solved that problem (...) you end up again porting that data from those disparate SQL and NoSQL databases either to a warehouse

That's exactly how that problem has been solved successfully for the past 20 years.

View on HN

The high latency between operational data and that data being reflected in reporting using traditional data warehouse pipelines makes it difficult for companies to make effective business decisions in a fast-paced business environment. Even in competent execution, that latency is frequently measured in weeks for big businesses. In the last few years, I've been approached by a number of traditional big businesses looking to rearchitect their database systems to make them more monolithic for the express purpose of reducing end-to-end latency in support of operational decisions.

It is extremely expensive and slow to move all the data to a data warehouse. Ignoring the cost element, the latency from when data shows up in the operational environment to when it is reflected in the output of the data warehouse is often unacceptably high for many business use cases. A large percentage of that latency is attributable solely to what is essentially complex data motion between systems.

It is not necessarily the case that data warehouses have high update latency. Open source streaming tech (e.g. Apache Kafka, Beam, etcetera) can be used to build an OLAP database updated in near real-time.

Sure, I've done this many times myself. The big caveat is that this only works if your data/update velocity is relatively low. I've seen many operational data models where the data warehouse would never be able to keep up with the operational data flow. Due to the rapidly increasing data intensity of business, there is a clear trend where this latter situation will eventually become the norm. I've already seen instances at multiple companies where Kafka-based data warehouse pipelines are being replaced with monolith architectures because the velocity of the operational system is too high.

For it to work, online OLAP write throughput needs to scale like operational databases. This is not the case in practice, so operational databases can scale to a point where the OLAP system can't keep up. The technical solution is to scale the operational system to absorb the extra workload created by the data warehousing applications, but current database architectures are not really designed for it so it isn't trivial to do.

Our data is near real time. But I can see how latency can be an issue based on size of the data and transformations are needed on the data before it hits the DW. But there are solutions - kafka being one.

Well, it's debatable if that is "solving" the problem or just hiding or mitigating it.

Reply on news.ycombinator.com