Building resilient services at Prime Video with chaos engineering
aws.amazon.com
aws.amazon.com
They have (in my opinion) an extremely good library of articles in their so called builders library.
For example these two below articles.
https://aws.amazon.com/builders-library/avoiding-fallback-in... https://aws.amazon.com/builders-library/leader-election-in-d...
These topics are extremely hard to solve from scratch yet they are distilled pretty well in the above articles and they include a further reading section.
I would implore others to have a gander. I wish the same could be said for their documentation.
The aspect ratio on the thumbnail for film I watched was stretched the other day, which is not the lack of attention to detail I'd expect from such a rich company.
Apart from this, it just isn't a smooth experience to navigate or search in. As usual in development nobody is paying any attention to responsiveness.
There is no reason this app shouldn't work as well as the youtube app in these respects.
A shame considering the amount of money they have. You'd expect a bit more, but then again Amazon are in the business of scale, quantity over quality.
At least it keeps them from having a monopoly on literally everything.
There’s an innuendo in there somewhere...
https://www.pagerduty.com/blog/outage-post-mortem-june-14/
Note that pager duty now weathers Amazon outages beautifully, but the point is that amazon doesn’t have 100% uptime and that properly designing distributed systems is very hard.
Why would anyone do that?
Prime Video has 4k videos and nobody else can use wifi if I play an Amazon Original and they forcefully opens in 4k and I take up all the bandwidth
It’s also better than one that always says I’m logged out when I start it, and then magically logs me in a minute or two later... while I’m in the process of trying to log in. (Or any of the several other ones that require me to log in nearly every time.)
Of course calling any of these a “User Interface” is almost laughable. SLOM is a better term. That’s “Small Library Obfuscatory Method” for the uninitiated. Or maybe they’re just trying to save bandwidth by making viewers spend viewing time searching pointlessly through their offerings that are ordered seemingly according to the pattern of pigeon droppings outside their office.
I still remember the time when Google tried to license their search servers to enterprises. It appears that market has now been completely eaten by Algolia. And my hunch is that it's because Google's search results are completely irrelevant for professional users. I search for a Windows API function name, and I get pages of SEO spam trying to sell me an unrelated Udemy beginner's course.
- When you search for a title, even if that title is in their library, odds are the exact match won't be the first result
- The differentiation between what is rentable, purchasable and freely streamable (with gradations based on different subscriptions) is incomprehensible from the listings pages.
- they split their shows by season to make their catalogue look bigger, but it's more confusing for the user
- don't make it explicit on the thumbnail either, so you have to hover over it to know what season of the show we're talking about
- although they do a decent job at continuing to watch episodes in a season, they're not making it clear when you've already watched entire seasons and those remain on the screen and are recommended to you over and over again
- advertise a trailer as an entire new season (see: The Boys season 2), leading to pretty gigantic disappointment (vs the netflix experience where a red notification for something you like is actually a great surprise because you KNOW it's an actual release)
- have a lower chance of being able to reload a page of something you left mid-watch and the page successfully reloading to the right content and timestamp
- the overflow of their preview hides the show underneath the current one which means any directly vertical scrolling is infuriating because you will trigger the overflow of the show underneath (that you see 20% of) only after having gone vertically down beyond 80% of the div in question
- the line = product line thing is confusing, as everything starts with "prime" for a full screen, (so why the repetition? oh, it's because ->) suddenly you get news (I don't care about them) and movies to rent or buy (I also don't care about paying prime to pay on top of it)
Of the 3 services I subscribe to, only Amazon gets things so wrong so often.
What I'm instead seeing is an attempt at copying Netflix' interface. This is actually far too common - competing services that just copy their competitor's homework when it comes to UI.
I mean it makes sense in a way if you're targeting that service's current members, they're familiar enough with the UI etc.
But TBH I would just like to watch one video on prime without it being stretched out of shape. It is probably some weird edge case with my particular STB, but every other streaming service seems to manage it properly.
Someone clearly wanted to do something "different and more beautiful than Netflix". The result was basically scrolling through shows one at a time. search was bad. Horrible.
My wife and I tried to re-watch some episodes of Brooklyn 99, and every time we would watch them they would start 20s from the end of the show. Like they stored that we had last watched ending at the end (-0s) and now when coming back months or years later, we might want to resume watching the last 0s with a buffer before (a bad resume behavior).
It seems they finally dropped that UI I'm the last few months to something sane, maybe the same as before I don't remember.
One would hope that for the 10x price increase over bare metal, someone would abstract away things like the underlying hardware or OS failing. But apparently, no, you have to pay the premium and do all the work.
The only use-cases that I could imagine for such a tool on EC2 would be if you either don't use containers, or if you oversubscribe your virtual servers by having higher container limits than what the instance can endure.
In the first case, the proper fix is to use containers. Docker can do CPU limiting for you so that one service spiking won't affect its neighbors on the same instance.
In the second case, I'd go bare metal and then hardware is so cheap that there's very little temptation to oversubscribe on RAM or CPU.
Author of the article here.
The core concern is not about the capabilities of the compute abstraction being used (bare metal, containers or functions) or testing OS capabilities. The aim is to validate mitigations which are in place to counter turbulent scenarios (For example: massive spike in traffic, network outage, dependency is down, etc). These scenarios generally originate outside the given system.
These kind of questions should be asked and systematically validated (quoting the article):
* Have you tested how the system behaves when the underlying instances have a sustained CPU spike?
* Is the system behavior understood under different stress?
* Is there sufficient monitoring?
* Have the alarms been validated?
* Are there any countermeasures implemented? For example, is auto-scaling set up, and does it behave as expected? Are timeouts and retries appropriate?
"Have you tested how the system behaves when the underlying instances have a sustained CPU spike?"
Since dedicated boxes are cheap, I'd just buy 5x the CPU resources that I reasonably need and call it a day. If there ever is a more than 5x traffic spike, then docker will prevent it from being a noisy neighbor, so the affected services will just become slower than usual. But even a 10x traffic multiplier would just produce a 2x slowdown, which should be tolerable for most users.
I agree that on clouds you want to save costs by only booking what you need. But bare metal, you can usually afford to keep spare capacity around all the time.
As such, I wouldn't plan for the system to behave well under stress. I'd try to always have enough resources around so that stress never happens. At the end of the day, this seems like a developer time vs. resource costs trade-off and for most companies, developers are sparse and resources are plentiful, so they'll have a very different trade-off from big FAANG companies.
"For example, is auto-scaling set up, and does it behave as expected?"
If your system is usually 90% idle, I wonder if you'll ever need that auto-scaling. Also, I'd say my customers can endure it if page load time goes up from 100ms to 200ms. So in my opinion, there is little need for auto-scaling for most companies.
You didn't really address this question, you addressed a different question, which is a traffic spike.
>Also, I'd say my customers can endure it if page load time goes up from 100ms to 200ms. So in my opinion, there is little need for auto-scaling for most companies.
100ms to 200ms average? What about the tail? Your app might go from P99 - 500ms to P95 - timeout. That's when you'll lose customers.
Well, with the 2x example, my app might get from a 1s P99 to a 2s P99 which feels slow, but is still doable. Again, those timeouts are usually introduced by cloud infrastructure. For example, if you use nginx outside of Heroku, it won't have a 30s timeout for file downloads.
Even if you're running on bare metal I find it hard to believe you don't have a layer with short timeouts between your front and backend.
The article talks about injecting failures at various points in the system, understanding how the system behaves under this stress, putting in counter-measures for the resulting problems, and eventually re-running this to validate those counter-measures.
In the virtual cloud world, that is common, because you rent the cheapest instance that will be big enough. In the bare metal world, that is rarely the case, because you usually get a Ryzen with 16+ cores and 128+ GB of RAM. In that case, there's no point in checking what will happen to your 200 MB web app if there's a CPU spike. It'll be just fine because the hardware can handle 10x the load without a hickup.
Similarly, if your page load time is dominated by internet latency, it doesn't matter if your CPU needs a few more ms to spit out the page HTML. So there, a 2x CPU usage increase will be barely noticeable to the user.
This isn't the case when your baseline is 6-7+ figures worth of machines
- You operate a service with significant traffic, and through empirical experience, you have a good handle on what 1x traffic looks like, and have even seen spikes to 2x traffic on rare occasion, which your overall system handled just fine. Applying your overall philosophy, you setup your system to allow for 5x the CPU resources you need, and call it a day, nothing to see here.
- But, guess what? Unbeknownst to you, your system has some critical bottleneck that would only surface at 3x your usual traffic, which could be anything from hitting some misconfigured max limits on your load-balancer, or exhausting all your database connections, or running out of threads or inodes on your server hosts, or triggering a kind of retry-storm/brownout due to slowly increasing latency in one of your service calls that only explodes past a certain limit (due to some unintended interaction with your core timeout/retry logic), or any number of latent potential bottlenecks that you never knew about, because as long as your system stayed under the critical limit, it was completely invisible to you. In other words, these are non-linear failures, that you cannot simply solve by extrapolating out with "1x traffic = 1x # of servers, 5x traffic = 5x # of servers".
- As a result, not only do you don't have nearly as much head-room for scaling up as you think you do, but ALSO when you do encounter such a failure, you cannot easily just "scale out" horizontally, because the failure mode itself is only exacerbated by horizontal scaling. When you encounter such failures that break some axiomatic assumptions you have about your system, it can be incredibly difficult/painful to reconcile, especially if you had no plans and no knowledge about these invisible/latent aspects of your system ahead of time.
Chaos engineering isn't about scaling at all, not really. It's about finding latent defects in your system, by actively probing your assumptions and seeing if your system behaves as you would expect. Using traffic to generate stress on the system is just one way to introduce some "chaos", but there are many other ways too (as covered in the article).
Of course, it's also true that systems need to reach a certain minimum level of complexity, before the ROI of introducing chaos engineering becomes really worth it. You need to have a complex-enough set of services, dependencies, or interconnected components that are likely enough to behave in non-obvious ways, that you have to do independent chaos engineering to test them effectively, rather than simply reasoning about their properties directly.
My experience is that I have yet to work with a company where this level of failure-proofing makes financial sense. Purchasing more hardware than needed is relatively cheap for most medium-sized companies, and it provides a fair level of protection against outlier accidents.
I'm aware that many people using cloud also ascribe to the 100% uptime mentality, but for most companies that is simply not needed. I mean even for Netflix or Amazon Prime Video, I wonder if 2 hours of unexpected downtime per year would really be enough to make anyone cancel their service. I myself at least have spent much more time than that trying to get HDCP graphics cards drivers, HDMI cables, and the stars to align so that the Netflix app will work with 4K HDR playback on my TV.
So yes, (your 2nd paragraph) I would knowingly accept that there are critical bottlenecks that are unknown and that could be triggered by severe traffic spikes. And most of my customers would be happy to accept that risk in exchange for the cost savings of not proactively fixing the issue.
And if you look at the overall state of software, it looks like pretty much every company is happy to trade reliability/resilience for cost savings these days. That's why I applaud the efforts in the original article, but the pragmatic way seems to be to just skip the whole thing.
That approach seems like a good idea, even if you think you know what the failure modes are.
But also this 'new field' just seems like something we've already been doing just with a different name. You're kind of expected to make sure your system can work if the computer suddenly shuts off, or a dependency is lost, or the network is slow. Have we not been doing this??
Sure, part of it is trying to seem cool and whatnot, but that's fine with me. The output is the same, more knowledge shared.
To avoid more of the same, I’ll say that the article itself is pretty detailed, but whereas Netflix blog-posts are more general, this one feels hyper-specific. That’s great if you’re tied to aws, but I think the former has longer-lasting + further-reaching utility.
Hope they do more.
It's an anti-marketing sentiment.
Big companies aren’t writing the doc, per se. Engineers interested in the material are. And often, they want the opportunity to do so - it’s something of a stamp for “expertise” in their favor.
Assuming the post is actual quality, it’s a marketing win for the company, a professional/developmental win for the employee, and an informational win for everyone else.
sure I can skip it, but why should I have to?
Please take a look at the underlying library here (AWSSSMChaosRunner) - https://github.com/amzn/awsssmchaosrunner
Turns out people care only if there is an end user impact and doesn't really care about random anomalies.
And to build the capabilities required for measuring the impact + automating the workflow of the actual chaos tests is a lot of work
A library like AWSSSMChaosRunner would be a core component of building gameday like capability. But building a full gameday framework is out of the scope of this discussion.
It worked a treat for ensuring high availability in IoT systems
(Not to mention the cases where it says everything worked but it didn't)
Injecting specific error codes or exceptions is a bit more complicated but it is possible with other approaches, for example: Chaos toolkit.
Doesn’t that sorta defeat the point of “chaos” a bit?
The AWSSMChaosRunner approach can't be used for Lambda because of what @vasco said.
You can take a look at a different approach here for failure injection in Lambda - https://medium.com/@adhorn/failure-injection-gain-confidence...
if I use prime video in a browser.. it works fine..
I have no idea why it does this and support has no clue.
There may technically be a license attached to them, but there is no practical way for any company to revoke my usage of them and the failure mode if any of those companies cease to exist is for them to continue working.
> Just remember anything you purchase on Amazon Prime, you don't practically own it.
I liked that Amazon reduced the price by almost the amount of the first two videos that I rented.