AWS faulty UX cost our business $7k and they refused to do anything about it
mikemorgenstern.medium.com
mikemorgenstern.medium.com
Is the argument that this should have been a pop-up that required you to click “ok” instead?
"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."
I also review my bills every month, which would have mitigated this issue significantly.
A cursory look would've been enough, really. If this would have been 20$ or so, I would not fault anyone for missing it. But 800$ missing every month should definitely get some attention, especially when these are amounts endangering your ability to stay afloat, as the author seems to argue.
Admittedly the feature that Azure has that AWS doesn't is "kill everything when exceed X dollars". Not so great for prod, but superb for dev.
Please realize that when you hand devs with AWS keys, you're assigning an unlimited liability to your company with whatever accident a dev might do. (Accidents happens, and quite often when experimenting.)
I'll elaborate: human error is not all reducible to a UX problem, at least not in any meaningful sense in casting blame at AWS here for causing the user's error. Giving users an explanation of their actions at the top of the page is not particularly bad UX, it seems adequate. If course they could do more, modals that say "here's your estimated costs for the action you're taking" and that might be better. But then again that might not be practical across the range of AWS modules, and a user that if ignores their bills and bank statements for most of a year might easily ignore those notices as well.
If you define every mistake a user makes as a UX problem then you've defined it broadly enough that it loses meaning. I doubt that's you're overall intent, but in claiming the user's failure to read the top of the page was a UX problem you're crossing over into that territory.
I might be more sympathetic if the user wasn't so lacking in awareness that they also didn't read their bills and bank statements for so many months, and didn't take the hour of time upfront that was ultimately necessary to understand part of a massively complex system. No reasonable level of UX design would have fixed all of that.
It's also a bit ambiguous-- it seems to be focused on how it will change the settings and last modified date. I could see how one could read that as "this action will create a copy of your files [and then delete the old ones] rather than moving them directly", warning you that your file metadata will be impacted.
There's also the problem that these warnings are everywhere in AWS console so you develop a bit of blindness to them.
So they have a pretty much wrong button name ("Edit Storage Class" --> "Copy to a different storage class") with an ambiguous warning message commingled with a bunch of other warning messages. I'd call that bad UX.
That being said, 8 months is too long for this bill to be considered AWS' fault. and the fact that he's been working really hard on his movie is irrelevant and comes off as whiny
Plus the following. UX is the umbrella term for "User Experience", not User Interface.
Therefore I'm not defining UX as users making mistakes. I'm defining UX as the user's motivations, journey, and actual experience with the site (and possibly extending to their interactions with the support team). This is not an 'overly broad definition', this is THE definition of UX.
If anything, what's happening here is the reverse. People are defending AWS' bad user experience, by narrowing down the definition to the narrow scope of UI-design and safety-nets. However, no amount of UI safety netting will salvage a website with bad UX (or rather, where UX never came into play during the design stage, except at the very late stages when it was only down to designing a UI).
So to reiterate. I'm not blaming AWS for the user's effective failure to read a technical warning message which then caused them harm. I'm blaming AWS for the fact that it even came to that in the first place. That is bad UX.
If it doesn't operate like Dropbox under the hood, don't make it look like Dropbox in your interface.
If an action creates a copy of objects, call it "Copy Objects" or whatever.
I think that presenting an S3 bucket as a set of files that you can select and perform operations on sets up false expectations here. And not showing incomplete multipart uploads in the file listing or having any way to see they are there isn't great.
Also user experience follows pretty heavily from UI design, you cannot separate the two without losing something. But "experience" is also much more subjective: Some might have a bad experience digging through menus while another person might have a good experience, like the way it's all organized. As an example, I found the notification for this issue perfectly acceptable: it let the user know a copy would be made. That is a pretty clear indication that the originals will still exist, and new ones would end up in Glacier. My experience there would be fine.
User experience is unhelpful in this respect because you cannot design your system to satisfy the experience of every user.
This is 100% negligence on their part, not AWS's fault. The cloud isn't magic, you still need to read documentation.
When you did the glacier operation, the GUI was telling you that it was COPYING the files to glacier. Those files also existed elsewhere, in normal S3 space. There's ways to cure this issue at the command line, primarily a list of all attributes per file - and there you can see the respective storage plans.
Secondly, do you have billing alerts set up, at 1.25x per average day's accrual? (30$/mo would be around $1.25/day, as not to email-bomb you). Billing alerts would NOT have solved the issue of improper storage class, but you would have seen a massive spike in costs. It probably would have cost you maybe $100-$200 to solve it early on.
Thirdly, how much of the 5 Pillars of Well Architected Framework are you following? ( https://aws.amazon.com/blogs/apn/the-5-pillars-of-the-aws-we... ) Admittedly some of these are "pay AWS double and get disaster prevention" style rules. However... pillar 5 is all about cost optimization. An ounce of prevention is worth a pound (or $7k) of cure.
My assumption when reading that message was that it would appear to update the storage class, but actually create a copy of objects with a different storage class and delete the old objects. It doesn't make sense that it would just duplicate my objects, and I still can't really imagine anyone wanting that. Why would a box called "Edit storage class" create a copy and then not destroy it? Why not make a dialog box that just...edited the storage class?
So maybe I had a previous version of all my files sitting in regular S3, with the current version sitting in Deep Glacier, and I may have just happened to delete previous versions of objects when I created the lifecycle rule.
To answer your question -- I don't think we're following any of the pillars of well architected frameworks. I'm a filmmaker who knows Javascript and PHP and not system architecture, and I was looking for a cheap way to not lose our movie to the ravages of time. I'm definitely not the only one.
> create a copy of objects with a different storage class and delete the old objects. It doesn't make sense that it would just duplicate my objects
Isn’t that exactly what 'copy' means in pretty much all IT contexts?
It seems that some "move" operations are "copy" operations, which is the context in which I understood this warning:
https://forums.aws.amazon.com/thread.jspa?messageID=455101#j...
The author can argue that the UX is bad, but they can't say it's "faulty" or "inaccurate".
Edit... After fully reading the article, the author even says this: "If an interface doesn’t quite do what it appears to, it should give a warning." The warning is right there in the UI, captured perfectly by his own screengrab.
Which is presumably why the article mentions the below in the second paragraph.
"After this experience, I cannot honestly recommend using AWS unless you have a professional sysadmin operate it for you"
"Sure, what do they say?"
"Who?"
"The errors"
"I don't know, I clicked them away"
A UX that has a high probability of causing harm is accurately described as faulty. I'm with OP here, technical warning-box being present aside.
The charges were caused because (a) selecting all does not select incomplete multipart uploads, which are hidden in the interface, and (b) I believe the operation is not recursive.
If it created a copy of the object as described, the charges would go away.
The warning may have well just said please note that the combothrombobulator may break existing encabulator nodes. If what you need is duckbill-style joints instead, use a thermocoupling-paradigm instillator instead.
When it should have said "THIS MAKES EXTRA COPIES WHICH COST MORE".
Its essentially saying I'm busy and negligent. Now I'm mad big company didn't come to my rescue.
> scaling our business from 3 to 10 and then 100 people, raising money, editing our film, and planning our release. Oh, and a virus shut down the world, I moved two times, and our business went completely digital.
> and amid 18 hour workdays,
The quantified communication with company provides no details to what was said.
Maybe op was right, maybe they weren't but this sounds like whining without any evidence.
If OP didn't notice it for 8-9 months, I do think it's unfair to blame the entire amount on AWS.
> Messages often took a full month to get a response.
Also does not ring true to my experience with AWS unless the author was not paying for any support tier. If they did not have even developer support on the account, then they got exactly what they paid for.
Maybe you should step back and think why people like us put our blood and sweat and it still takes years to learn how to administer infrastructure properly. Stop whining and learn to pay for skills.
The comparison the prior comment made was something along the lines of "both work in very similar setting and industry, but each have their own skill set and expertise." Each can probably function in the the others job, but not well and probably at a greater cost.
Just because the current status quo of things is this way does not mean it is great. The world would be a better place if more people were empowered to setup computer systems in a safe way.
But I've been in this business for a while. I'm a programmer rather than an ops person, but i still sometimes make changes like that. I am responsible for some S3 buckets of stuff whose cost is significant for our budget.
I would never make a change in any software, where a failure for the change to take would cost me a noticeable part of budget, and not do even a cursory check to see if the change took. I mean, I write software, I know how it goes.
Who does that without ever checking to see if it worked? I don't do anything that matters without double-checking to see if it worked like I expected, at least if I'm doing it for the first (or second, or third) time, whether via a console or a script. Cause if you work with computers, you know they don't do what you expected all the damn time.
You're making a new kind of change you have never done before? You try it with a few objects before trying it on your whole bucket. You double-check if it did what you expected. Then you do it to everything. Then you still double-check to make sure it did it as expected to everything. (Even if you "read the manual" as the OP says, it's not about that at all in fact). That's how engineering works. If you don't have time to do it carefully, then sure, your chances of making mistakes or misunderstandings goes up a lot, why would you expect someone else to be responsible for your mistakes from not being careful?
It also speaks to poor attention to detail when a small company starting out can accidentally overlook ~$1000 per month in unexpected charges.
I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day. I don't expect them to step in like that for a problem sustained over the course of months.
This. I manage a 15k/month AWS account, and if the expected expenditures of any day are > 15$ off the normal, I try to find out what happened. Especially after changing stuff in S3 and EC2; as with our usage patterns they should have predictable billing.
> I expect AWS to maybe help out if someone fat fingers a number and, I don't know, accidentally spins up 1000 m5.24xlarge instances for half a day.
I've once reserved instances in the wrong region, but (luckily) found this out the same day, and got it refunded the day after. Would have been the equivalent of a year of salary down the drain otherwise.
We had a run away EFS volume that burnt through $300 in a day, luckily was caught by an alarm. Thats basically $8700 saved if accounting had caught it first.
We were unaware that changes via a rule still count as an api hit per object, despite all happening in the backend. I’m sure we can’t be the only people hit by this.
And yes, it's both fairly well documented and something that can catch you by surprise at the same time.
But that's just my opinion. Maybe others like buying things from a menu without prices.
DigitalOcean emailed me once when I left my single $5/month instance open by mistake. AWS would never do that even for thousands.
Would it be so hard for them to redesign AWS to tell you what something's going to cost before you do it?
However, given the nature of cloud services in general (they have a strong $$$ motivation to nickel-and-dime you for every little thing they can) and especially AWS' reputation for doing that, I likely would not have made this mistake. Then again, I'm also someone who carefully reads all available information --- especially whenever it's something I'm paying for.
While the cloud is just someone else's hardware, you still pay for it like it's your own.
One wrong click and you can blow your budget, crash you entire system, the works. Over the years I've grown used to it but the feeling of uneasiness will never leave me. I get it, AWS is very powerful but it does not give me the comfy feelings of using a product like Render or DigitalOcean, where I pay for what I need and that's the end of it.
This tutorial includes how to setup alerts for a given budget: https://aws.amazon.com/getting-started/hands-on/control-your...
I don't believe you can "halt everything" if you go over a certain amount though. I'd be happy to be proved wrong.
> I don't believe you can "halt everything" if you go over a certain amount though.
You're pretty much on point. Handling a hard cut is simply not feasible for nearly all services. I.e. simply sending E-Mails in the middle of a news letter is not useful. You can have reasonable actions for some services (i.e. disallow new EC2 machines, add the end-of-month-cost for running ones to the bill), but this could break automation in strange ways (i.e. you suddenly could not spin up a large instance temporarily due to the limit). And how would you handle traffic? Disallow peaks? What about API gateway or other usage-based services?
Lastly, AWS is all about scalability. Breaking so much, especially in the scalability space, for such a small feature is simply not worth it for them. So usually they're simply lenient on bills and take the occasional beating (and it's not like there would not be an equal amount of articles on "how AWS caps broke our system [and it did not warn us/the UI was bad/ ...]").
The missing hard cap is the reason I don't have an account there, but I can absolutely understand why they don't have it.
I've never been in a situation where I wanted it decreased, but I guess you could setup an account and ask for a decrease.
If enough people ask for it, they might make it something you can do yourself.
What's different is the perception that cloud infrastructure should be significantly easier. But the ways in which it's easier are not in the area of mitigating the impacts of human error.
We also know dev work, but it's not our specialty. We also know how to manage projects but we're not project managers. We also know how to handle billing, but we're not accountants.
... But our job encompasses all of them. And we reach out when we don't know. Our job is to make the trains run on schedule, using the average amount of fuel, with nobody stressing, and the customers happy. You get rid of us.. and... $7k of overage is nothing.
Instead, I'd look at contract system administration labor. It'll probably set you back 100-150$/hr with minimums of 2h. However, sysads can catch problems, set up alerting, and fix structural issues before they turn into humongous problems.
It would be my recommendation to do more hours with onboarding a consultant and a full audit (not SOX or FedRAMP or the like - but to get you aware of what you're doing even if you weren't aware). From there, a plan can be made to keep you up and running with low/no impacts. And once the hard part's done, should insulate you from any oopses. And the monitoring will help catching them in the act in case there's a new type of oops.
I hope I answered you adequately :)
Given that I was pretty knowledgeable about most aspects of backing my data up to S3, it would have been pretty hard to predict where the possibility for error would have been. I can't imagine hiring someone for 2hr to come and check my work, since they really would have had to be there for the entire process.
It sounds like about 2-5 days of work to get someone to set all this up, but perhaps it's more like 1 day at a minimum. So somewhere between $800-4k for that. Later in our project we were running 5 S3 buckets with about 20 stakeholders uploading and downloading, so that could have easily been another 4 days of work.
$800 would have been two years of what we were planning to pay Amazon. $4-8k would have been a major big-ticket item for us (even though, of course, that's the bill we ended up stuck with.)
If I were to do it over again I don't think I'd be able to justify spending on a pro. I would definitely spend on someone to audit our expenses more carefully though.
But he waited 9 months and kept paying. That's just too late.
Honestly, even as a quite technical person, understanding AWS is not that easy. I've been in the situation of trying to run a very small app with basically zero budget on AWS a few years ago, and the sheer amount of services I needed to involve and things that needed to look for where quite overwhelming. I can easily imagine a non-technical person getting completely lost in the interface.
Now there are 6 different storage classes and loads of options to manage the objects and buckets.
Interestingly the latest storage class(S3 Intelligent-Tiering) was created to minimize the overhead of S3.
On AWS, I know how to manage IAM roles, set up EC2 instances, etc. etc.
The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.
I agree with you about this, certainly:
> The idea that only a dedicated IT professional should be using this product really cuts out a lot of people who can't afford a dedicated IT professional.
But this is also like saying "You should have hired an expert to tell you that when the sailboat builder kit manual says drill into the back it actually means the front." Regardless of whether someone more experienced at this would know better, this UX is awful, its title is inaccurate, and the whole process around changing storage classes far more confusing than the UX makes it seem.
There are petabytes and petabytes of data from independent filmmakers that are rotting away on drives, and part of the issue is lack of other affordable options. So I get that "shoulda hired a sysadmin" is an easy response but that's not always an option, especially when there's dozens of hard drives or other complexities that make these tasks last weeks or months.
Amazon has traditionally been good about waiving charges for account compromises and things along that nature. However you do have to keep up on it since they bill you every month. If you're running a company part of the responsibility is at least checking in on your financials every so often. Right when you see a charge for $700+ when you expect $50 is a red flag. At that point if he had contacted Amazon I'd be surprised they didn't waive the charge.
I'd be curious if Amazon was willing to waive any part of the charges at all.
A small business should use a backup service like Backblaze. AWS is what you use to _build_ a backup service.
Backblaze, last I checked, has "unlimited" storage but requires you to plug the hard drive in every three months. A number of other details made it seem like it was not a good long term storage option.
I was focused on a solution that would let us store the film we spent a year working on for the next 50 years. We did this and LTO tape.
They say things like this with a straight face:
- The cloud is always cheaper than on-prem!
- The cloud has no downtime, unlike on-prem!
Recently, I came across a large government department that had moved a reasonable volume of stuff into Azure. Think 7-figure annual bills.
They didn't like that they were overspending, so they complained to Microsoft, essentially demanding that they "make good" on their promise to make the cloud cheaper by cutting the cost on resources. This would have required an 80% discount, maybe higher. It's an absurd thing to even ask, but ask they did. Microsoft said no.
Meanwhile, I looked at their tenant, and the stuff that goes on there is amateur hour: Daily uncompressed database backups kept forever in premium zone redundant storage. Dozens of misconfigurations like that just burning money.
The worst part of it is that the "Azure Advisor" tab on the portal literally calls this kind of stuff out, recommending fixes worth hundreds of thousands of dollars a year, many with a one button "Quick Fix!" that you can press for an instant discount.
Nope. No. They're not going to bother reading things, or checking the Cost Management portal every now and then. They'd rather complain to their Microsoft rep every few months in a huffy tone.
"You still haven't cut my costs! Why haven't you!? I thought that's what you people did!"
They're busy people and don't have the time to waste cutting half a million dollars of spend here or there with arduous tasks like button pressing.
An consultant specialising in AWS once tried to convince me that moving our large scale computation into the cloud would reduce the environmental impact to nil.
Amazing what wild theories people will come up with when it's hidden behind a curtain.
More like "...when it helps Amazon make more $$$."
Which is basically what everyone who has "drunk the Cloud-Aid" will be doing, not only for AWS but other cloud services.
We're sorry that this happened and hear about it a lot. I'd highly recommend folks sign up for Vantage if for no other reason than to get our weekly cost reports which would have caught this sooner: https://www.vantage.sh/
This article should have been written in a less bombastic fashion, I felt at the end like it was just an advertisement for a film.
Sorry you didn't like the tone of the article.
And seriously:
> This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule.
I hate the AWS Console UI for a bunch of reasons, but short of making you type that sentence before you proceed, I'm not sure what AWS can do here.
But an estimated cost for many things in the console would be a welcome sanity check.
I didn't suspect that because in all my correspondence with Amazon, they always said the issue was incomplete multipart uploads.
So I assumed, based on the dialog box called "Edit storage class", that the net result of this action would be the same files but with a different storage class. This led me to read the warning as "We will create a copy of your objects and delete the old one, so please be aware you will be charged for the cost of the copy" rather than "We will create a copy of your objects and two versions of them will exist forever."
So it does seem like Amazon tried to warn people. But why on earth would a dialog box labeled "Edit storage class" not do anything of the sort? Are there actual legitimate reasons someone would want to have two versions of their objects sitting around?
Say what you will about my billing awareness (and you are saying it. Feeling the hate, all, thanks) but this is not good UX.
If there's any legitimate reason to duplicate objects in the name of changing the storage class, I'd love to hear it.
First, the same warning you are referring to also says, exactly: “You can change the storage class without making a new copy of the object using a lifecycle rule.”
Second, to answer your prompt, the reason why the object is duplicated during a storage class change is because S3 is an object store and not a filesystem like you may assume. Performing a modification operation on an object does not change the object itself; instead a copy is made with modifications. When changing storage class, a copy is thus made. If you had used the lifecycle rule as the warning suggested, it would have performed your assumption: make the copy and delete the original object in the background.
P.S. Not trying to be abrasive :) I do though think you should considering hiring a real devops/cloud/sysadmin to manage AWS for you. If that is really not feasible for you, I suggest at the very least looking into setting up budget anomaly detection and alerting. You then might be able to have some visibility into your budget.
Re your first note -- because I assumed that this operation was being treated similarly to a move (which is a copy and delete in S3), I assumed that I would be dinged for the cost of copying (minimal when on the same location) but the original would be deleted.
Reference: https://forums.aws.amazon.com/thread.jspa?messageID=455101#j...
Second -- this makes sense to me that this is how it works. But I still don't understand why anyone would have a use case for this "Edit Storage Class" dialog box. Is there ever a moment in the administering of an object where you would want to duplicate that object but with a different storage class?
Unfortunately hiring a sysadmin is way out of our budget. We had a friend helping with some AWS stuff and they may have done some of this for us if I'd asked, but I do remember it being a fairly mind-numbing task and he wasn't co-located with the files.
"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule."
"This action applies to all objects within the specified folders"
I'd assume the first one means youll get charged once for standard and once for glacier
The major lesson: audit your expenses every month (or hire someone to do it for you). The AWS UX cost them $800. Faulty business administration cost them $6200 on top of that. It could have just as easily been $MAX_CREDIT_LIMIT.
I don't think you understand what UX means.
This shows how incompetent this guy really is.