Comment by jedberg | Hacker News Reader

Comment by jedberg | Hacker News Reader

This was a big thing for us at Netflix too. It was extremely rare for all of Netflix to be down. Almost every outage was a partial outage.

For us to measure our success in increasing availability, we first had to figure out a way to measure availability.

We came up with a multi-pronged approach. The first thing we did is figure out how to predict how much traffic there should be at any given time. This was basically using historical data to determine the shape of the curve and then adjusting it to fit current traffic.

Then we would figure out how far off of the predicted traffic we were and the was our downtime.

Since we also had control of the client experience in almost all cases, we were able to measure from the client side as well (and trust that data), and include that data in our measurements.

Where things got interesting was when say a large ISP was down. A bunch of people couldn't get to us, but that wasn't really our fault (or was it?). At the end of the day the users didn't care if it was our fault or their ISP, so we counted that against ourselves.

All of this is to say that yes, it's really hard to figure out uptime, especially for distributed systems where almost every failure is a partial failure.

But at the end of the day your users don't care which microservice or network segment was at fault, they only care that they couldn't use the product.

>> Since we also had control of the client experience in almost all cases, we were able to measure from the client side as well (and trust that data), and include that data in our measurements.

The W3C Network Error Logging specification now allows websites to register client-side errors (DNS, TCP, TLS etc.), record them offline, and send them via side-channel telemetry to a different endpoint later on. This has changed how we measure client-side availability on browser-based services.

Chromium-based browsers (68+) support this feature, enabled by default.

>> Where things got interesting was when say a large ISP was down.

NEL also helps the service owner determine with some level of accuracy whether a problem in an ISP is specifically related to the service, or to the ISP

{disclaimer: work for MSFT, specifically on NEL-related tech}

[1] https://w3c.github.io/network-error-logging/

How do you guarantee that the client hasn't modified the telemetry data?

I doubt you can, but why would they? I mean, yes, maybe an unethical competitor could use it as an attack, and yes we live in a world where AdNauseam exists, but I would be surprised if that kind of thing was enough to matter at any scale (i.e. it'd disappear below your noise floor unless it was a really good mass attack).

Well, it depends on the use case. At Netflix for example, it was the outliers that were interesting. There aren't many 2011 LG TVs out there, but if they were all consistently failing, that would be something we'd want to call out.

Even every 2011 LG TV failing at once falls below the noise floor, unless you're specifically looking for it.

But if your are specifically looking for outliers, than a single bad actor can really mess up your day.

I don't think a guarantee can be made that the client can't tamper with it - its just an HTTPS POST request to an endpoint with a JSON body that's generated from the browser.

However, if NEL is run globally, it just becomes a part of the noise floor (someone else mentioned this as well), and to do effectively with malice will require doing this 'at scale' (that fav. phrase). So far, we haven't seen changes in the outcomes from telemetry analysis.

One way of removing malicious noise could be to apply the same bot-detection capabilities for NEL that you apply for the service its monitoring.

> A bunch of people couldn't get to us, but that wasn't really our fault (or was it?). At the end of the day the users didn't care if it was our fault or their ISP, so we counted that against ourselves.

So... what did you do about that?

Reply on news.ycombinator.com