Nice article.
>its not uncommon for some of our customers to send us millions of metrics every minute
What kind of customers/services generate millions of points a minute?
>its not uncommon for some of our customers to send us millions of metrics every minute
What kind of customers/services generate millions of points a minute?
Of course, our metrics were all handled in house. From talking to the teams that handled the metrics pipeline, the vendors were great for smaller companies, but there was no off the shelf solution for a companies that large with that volume. But I did very little with that myself, other than look into the fact that Spring Boot published way too many default metrics. Who needs P50, P70, P75, P80, P85, P90 - P99 on all web requests?! Just set a default that is small and worthwhile and let the developers adjust as needed.
If you're pumping out a million metrics per minute, almost none of those are ever going to actually be used to generate meaningful insight.
Agree 100%.
In most cases, time spent maintaining terabytes of rapidly aging time series data would be better spent elsewhere.
A particularly good high-frequency trader might be interested in Terabytes of minutia when they're trying to sort out what caused yesterday's spike and crash of ticker XYZ.
Systems and sales analysts that are looking at web store front ends (and back ends, if there are issues) would be interested in large volumes of data, specifically corner cases (users who don't follow a statistically significant path), when trying to sort out a UI/UX redesign.
Traffic and transit analysts might want terabytes of data (especially with date and weather indicators) when considering what kind of freeway interchange to add to a growing area.
I suppose I could go on...
(Meraki actually did implement its own time-series database, and after I left published a paper describing its design and implementation. https://meraki.cisco.com/lib/pdf/trust/lt-paper.pdf. Good quote on the motivation: "As discussed in Section 2.3.3, customers have a nearly insatiable demand for high-resolution historical data, even though they mostly query data from the recent past."
The example from the article was "one team at one of our customers decided to dump 30 million metrics on us, send all of their mobile product metrics into Outlyer"
There is virtually an unlimited number of applications that could generate 16k events per second (million per minute).
The result was a tens of terabytes a day Niagara of data.
When I left they were in early stages of Hadoop because ordinary parse/analyze was starting to fall behind.
Across their organisations it can be much more, Fastly has reported 2.2M/s (https://promcon.io/2018-munich/slides/monitoring-at-scale-mi...) for example.