Hacker News Reader

Comment by closeparen | Hacker News Reader

I work at a household name tech company and pretty much all our "data" dashboards are running SQL queries against a Hive/Presto environment that's at least 24 hours behind. Real-time dashboards are limited to the service metrics time series database (so combinations of categorical variables, no individual records) or rare expensive Pinot tables.

View on HN

I have seen this on the business side too. A lot of we need live data requests don’t really need live data. If the decision is made once a day, hourly data is more than enough. Yet we still build dashboards that refresh every few minutes just because people asked for it.

In this case, we do actually need live data! We use wide events for monitoring business events and it lets us understand our platform with utmost flexibility, which is important for us as a fintech company.

Traditional observability is ill-suited for observability around business events. What if you forget to instrument a counter or gauge for something? In my experience it's far easier to log wide events with as much context as possible instead of agonizing over anticipating the dimensionality of metrics upfront (you're going to miss something).

Perhaps changing is a better word.

Yes this was also my experience working at a large tech co. I work in fintech now and data volumes are low enough to maintain 2-3 minute up to a few hour data freshness.

Reply on news.ycombinator.com