Hacker News Reader

Hacker News

Comment by ilrwbwrkhv | Hacker News Reader

Full thread

ilrwbwrkhv

Looks a bit overengineered due to Netflix's own microservices nonsense.

I would be more interested in how a higher traffic video company like Pornhub handles things like this.

View on HN

> Netflix's own microservices nonsense

How many times has Netflix been entirely down over the years?

Seems it's not "nonesense".

Netflix's technology is so nonsense that right now for the boxing match they are dropping the quality to 720 pixels.

Have you worked with distributed counters before? It's a hard problem to solve. Typical tradeoffs are lower cardinality for exact counters.

The queue solution is pretty elegant.

The queuing reminds me of old "tote board" technology. I can't find a reference easily but these were machines used to automatically figure and display parimutuel payouts at horse tracks. One particular kind of them would have the cashiers' terminals emit ball bearings on a track back at the central (digital but electromechanical) counter for each betting pool. This arrangement allowed the cashiers to sell as fast as they liked without jamming the works, and then allowed the calculation/display of odds to be "eventually consistent".

How would you design it to support mentioned use cases?

HyperLogLog and PostgreSQL: https://github.com/citusdata/postgresql-hll

Or even simpler, Roaring Bitmaps: https://pncnmnp.github.io/blogs/roaring-bitmaps.html

https://blog.quastor.org/p/grab-rate-limiting

https://github.com/RoaringBitmap/CRoaring

HyperLogLog doesn't support exact counters though, does it? That seems to be one of the core requirements of the queue-based solution.

HyperLogLog (or even Count-Min Sketch) will not support some of the requirements of even the Best-Effort counter (clearing counts for specific keys, decrementing counts by any arbitrary number, having a TTL on counts etc.). For Accurate counters, we are trying to solve for multi-region read/write availability at low single-digit millisecond latency at cheap costs, using the infrastructure we already operate and deploy. There are also other requirements such as tracking the provenance of increments, which play a part.

Reply on news.ycombinator.com