The best way is to build small agents to monitor the service you depend on to know whether they truly respect their SLA. In case of LastPass they don't even have an SLA....so good luck with an updated status.
Magic, 100% uptime!
Their SLAs are short and readable. It is worth reading a few of them, especially if you have a SaaS and are thinking about your own SLAs. Just search Google for [google $product sla] to find them.
I know there is always the case where "it's just me", but I'm talking about an incident that was widely reported in the media because it was so widespread. While the incident is ongoing, they do provide status updates... but after a couple weeks pass, the global outage that affects everyone silently disappears from their archive. It's quite interesting.
There kind of needs to be a third party SLA escrow of some kind to really make things work.
The problem with this model is that it doesn't allow engineering teams to set realistic goals to improve reliability. Because a customer being down for 3 days doesn't cost the company any money, you can't deploy expensive engineering resources to prevent that sort of thing from happening again. Meanwhile, if you track your SLA accurately, and compensate customers in a way that's commiserate with the inconvenience they experienced (something like "the entire month is free if we're down for 8 hours in a row"), then you can start doing real engineering. You have a clear number that shows where you're at now, and you have a goal for where you want to be, and you have a cost associated with that goal... suddenly you can make intelligent decisions about what to work on. This class of outage costs us $600,000 a year. It would take one engineer at $200,000 a year 3 months to fix it. There's $550,000 of free money. Instead of being a cost center, you're a profit center! And customers get a better product. How is that not a win? I'll never understand.
One thing I liked about working on Google Fiber back in the day is that US-based telephone support was not something that we would compromise on. It was expensive! So when we could eliminate classes of problems that people call in about, like poor WiFi connectivity or bad TV remote Bluetooth pairing, you could directly see the savings in support cost. You could spend a year debugging WiFi, and instead of looking like flushing money down the toilet, it looked like making money. It was a joy to work on. But obviously a very uncommon way of accounting. It's easier to say "everything is perfect, we dare you to cancel" than to invest in engineering. As an engineer, that's sad; we want the world to work better... but it's only possible with resources.
Office 365 had an issue where their DNS resolutions were fubar and impacted the service for certain customers, but their position was that the service itself was fine.
In particular I'd see repeatable problems where they couldn't launch whatever browser X operating system in under 2 minutes (when our tests would time out) and list allocation time as 'elevated' (say, 8s average vs their normal of 3s). If you start believing your own statistics you get into almost as much trouble as believing your own PR.
For an 18 month period where they were particularly bad, I think they only copped to an actual problem one time out of around a dozen cases where our CD pipeline was blocked for half a day or more unless we just turned off e2e tests entirely.
What's a fair way of producing a single number though?
The relationship between client and service is the same irrespective of the number of clients the service has - a number which means exactly nothing to the client.
But more importantly, availability numbers are for informing a client about how much incidents outside their house can affect them, and reasonable courses of action to take when it does.
When using the numbers internally, the fudged number is equally misleading. Unfortunately there exist fewer adversarial relationships internal to an organisation to prevent these short sighted statistical nonsenses.
Maybe a more scalable solution would be a third-party company that sells this information. I think there’s a lot of money to be made there.
That's my conclusion on what status pages have become. Which of course raises the question: what do I do when I see a service with N problems in their status pages over the last X days? Are they being naive, or was their service so bad that they were forced to write it down?
I agree with you, there is a lot of money to make there. I think there are already a few companies doing that, though.