Folks need to resist the inclination to just gather maximum data for the hell of it.
If you're pumping out a million metrics per minute, almost none of those are ever going to actually be used to generate meaningful insight.
If you're pumping out a million metrics per minute, almost none of those are ever going to actually be used to generate meaningful insight.
Agree 100%.
In most cases, time spent maintaining terabytes of rapidly aging time series data would be better spent elsewhere.
A particularly good high-frequency trader might be interested in Terabytes of minutia when they're trying to sort out what caused yesterday's spike and crash of ticker XYZ.
Systems and sales analysts that are looking at web store front ends (and back ends, if there are issues) would be interested in large volumes of data, specifically corner cases (users who don't follow a statistically significant path), when trying to sort out a UI/UX redesign.
Traffic and transit analysts might want terabytes of data (especially with date and weather indicators) when considering what kind of freeway interchange to add to a growing area.
I suppose I could go on...