Hacker News Reader

Incredible what you can do with infinite money!

For everyone else, more specific data structures, sampling and careful consideration of what to record are essential.

The key is that you pay for the bandwidth, sampling, cardinality, and schema considerations somewhere in the system. Depending on the problem, you may be able to get away with dealing with those issues later vs. earlier, but at some point you start dealing with required fields, aggregation, etc.

I own a system in one of the tech giants where sampling is counterproductive to our problem domain, and where in the pipeline we deal with each of those issues is our bread and butter problem that can sometimes swing costs by $millions.

Hang on, you don't need infinite money if you've got a sampling rate, do you? Drop the sampling rate (mentioned in the article: they do use sampling) from 0.01 to 0.001 and you've reduced the data ingress by a factor of ten.

Reply on news.ycombinator.com