Problems with disk encryption in AWS
tmp.bearblog.dev
tmp.bearblog.dev
The entire point of encryption at rest (on the cloud) is that when any of the following happen you have nothing to worry about.
1. A machine/disk is rendered inoperable and can't be wiped.
2. The data stream coming off of a disk cluster is tapped.
3. An employee steals a disk or is lost.
4. An actor violates their account segmentation and can read raw data from segments of YOUR sectors of the shared disk.
5. SSD firmware goes bad/gets hacked and starts returning incorrect sectors of disk.
6. Memory pointers go bad and return sectors from the wrong area of the disk.
It's incredibly naïve to not use encryption at rest on AWS with how incredibly easy and problem free it is to deploy.
As I mention elsewhere in the thread, I use encryption, but I don't consider it to be of high value compared to other security mitigations one can spend time on.
As mentioned in the blog post, setting up encryption isn't always easy and problem free though.
Regarding 1, disks are physically destroyed if they are inoperable. Thanks for sharing the list of potential problems, it's always interesting to see how other people think and what they worry about!
'Supposed to be' is missing from this statement. We have seen many vendors over the years say 'we destroy disks' only to have a disk show up in the wild.
Where is it you draw the line?
Do you bother with Spectre mitigations since Amazon policy is to deny service to those who would attack you?
Do you bother requiring authentication on your database since policy is to use a closed VPC?
Hell, we haven't seen any man in the middle attacks lately, let's just drop SSL because you know your customers are wired and trust the endpoint networks.
SSL is the best example subject to the same common failures (including expiration, key loss, downtime to initially deploy) as you describe disk encryption migrations.
Encrypting disks is mandatory and there is zero justification otherwise.
I don't go out of my way to mitigate that, no. Have you seen any real attacks with this? They seem very rare and hard to execute, especially if you have someone specific in mind.
> Do you bother requiring authentication on your database since policy is to use a closed VPC?
Yes. Usually you have multiple services running in the same VPC, by using authentication you limit the potential impact if one service is hacked. Adding authentication to a database that previously had none is also very easy.
> Hell, we haven't seen any man in the middle attacks lately, let's just drop SSL because you know your customers are wired and trust the endpoint networks.
MITM attacks are quite common actually. Internally inside AWS (i.e. between instances) the benefit of TLS is maybe questionable, especially since traffic is encrypted between instances automatically (depending on the instance type).
> Encrypting disks is mandatory and there is zero justification otherwise.
I disagree that there is "zero justification otherwise." I've updated the blog post some and I'm interested to hear your thoughts about it. But in short, adding encryption to an unencrypted machine can take a lot of time and effort. Setting it up correctly from the start is usually easy, but it's not always whoever set it up initially did that. There are many things one can do to improve security, and most things will probably be more beneficial than re-encrypting disks.
It shocks me that this isn't on-by-default in AWS like it is on GCP.
By now that's moot so I'd assume they'll set it by default for new accounts at some point but there's a regional option which you can turn on by default:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncry...
If you use tools like Security Hub with the Amazon Foundational Security Best Practices suite of controls enabled there's a specific check for that setting:
https://docs.aws.amazon.com/securityhub/latest/userguide/sec...
I.e. how do you even know it's working and your data is encrypted? Just because API says so, doesn't mean it's so. There is no way for you to verify. Is there? Unless you encrypt on your end, you can't be sure.
I know people who work there. I've read the infrastructure designs. I know Amazon audits themselves. I know AWS has third party audits. I know AWS has government and enterprise users who require their partners be legally liable for improper implementation. I do not subscribe to conspiracy.
Also, sometimes there are mistakes with decommissioning old drives, and you wouldn’t want your data discovered in a landfill somewhere.
When I dug deeper, they had wiped the SMART data and the partition table, but it was absolutely full of readable data. I found clear text server logs indicating that this drive was in a backblaze center for several thousand hours.
> I'd worry they would sue for...
If you are referring to Backblaze, we're not going to "sue" anybody for anything.
We (Backblaze) have dealt with a bunch of frivolous lawsuits (and patent trolls) over the years suing us, and OMG we're not going to instigate any lawsuits over some honest person legitimately reporting some issue and being helpful. It isn't going to happen.
Our reputation is important to us. Not just for Backblaze: I'm saying the individuals that founded Backblaze and those people that work now here base our entire existence and careers and the number one marketing efforts at Backblaze are based around we are trying to be "the good guys" and transparent and acting like it. There is no possibly world where we try to suppress a screwup like this through legal means. That would be a PR debacle of epic proportions.
If something went wrong, let's shine a spotlight on that cockroach and figure it out together. I'm not sure the exact drive we are all talking about, but my first guess would be a customer ordered a $189 "USB Restore" all their data shipped to them on an encrypted drive) and we (Backblaze) shipped the customer a USB restore drive and they are subsequently selling it (after copying their restore off of it) on the open market. If it is above 8 TBytes this is absolutely *NOT* the case and we should get to the bottom of it. Without lawyers mucking up the situation.
What plausible damage have you done that they could sue you over? If all you want is your cost or the like, I can’t imagine that is a crime either.
If someone has a lot to lose, I guess sending a letter through a law office would be a sane option, but backblaze doesn’t strike me as the shoot-first-ask-questions-in-the-deposition type of company anyway.
If the government is out to get you, they'll try and find something to come after you for. Just ask Josh Renaud.
https://www.theregister.com/AMP/2022/02/15/missouri_html_hac...
I assume aws is similar
https://www.datacenterknowledge.com/archives/2017/01/18/who-...
Adding full disk encryption takes time from other projects and makes the system more complex. That equation needs to pay out. In all likelihood, the reason your data is going to get stolen is a privilege escalation in your app code or a bad actor on your team. Rogue AWS employee swiping your particular hard drive in us-east-1 is way down the list. Full disk encryption does nothing for the first two vectors.
I think compliance programs are oriented around pushing companies into complex/expensive system designs thinking that is a proxy for a secure system.
Usually you inherit an infrastructure, and it's usually not set up in this way (in my experience) and then there is a lot of work to re-encrypt the data in order to use KMS rather than the default key.
> it is typically standardized in an org
I have still not found any SCP I can set that prevent the use of the default key and enforces KMS. If you have one, I'd be happy to take it! If you mean "standardized" as in written on a paper, I'll rely on wishful thinking because people make mistakes or just don't know about it even if it's a standard.
Compliance has good sides too though. For example, they force you to think about areas your intuition might otherwise not have gone, so I don't dismiss them but sometimes it makes you spend time on less than optimal things in order to stay compliant.
This sounds like it's coming from someone who's forgetting that all of our data in "the cloud" resides in physical buildings throughout the world, which are all high value targets for physical attacks.
I don't think it's an antiquated idea to protect against physical data center attacks. It's best practice.
Could a data center break-in happen? Sure. Is it likely? No. When used in a data center, FDE is mostly useful when media is being transported or disposed, as an extra layer of protection.
I’m not saying it shouldn’t be used, but when comparing using FDE in other situations vs. a data center, in a data center the physical risk is far lower.
If they do, I trust them to be good stewards of their keys. If not, then why not? I don't know about the enterprise part of the market, but I know consumer SSDs come with encryption permanently enabled in the firmware.
Where is the encryption key? It seems that it must be in the firmware itself. Presumably it would be possible to find this with enough effort.
But in short, the key is kept in memory on the HSM, and employees don't have access to it. They key can be referenced, but not actually read.
It also means that if a user accidentally deletes their key, there's no recovery. That's it. (Pro tip: Deleting a key is a faster mechanism to make data unreadable than deleting the data itself. ;)
Disclaimer: I'm an S-TAM with AWS.
- the drive can request the key on each boot - the drive stores the key in the firmware, but part of the de-provisioning process would be to reset this key
The people with physical access in AWS datacenters are not the same people who have access to the encryption keys.
In fact, it's likely much more complex than that. The people with software access to the machines very likely don't have access to the key storing system (and definitely not the hardware).
This means that the number of people it's possible to bribe to get access to your data is much smaller than if you can just bribe a DC employee to either smuggle the disk out, or make a copy onto a thumb drive.
I'm not saying the number of people who could "break glass" and read anyone's data is zero. But it's at least an order of magnitude fewer than the hoards of people employed to swap hard broken drives all day every day.
And the people with this "root" access will likely be very well paid, reducing (but not eliminating) risk of bribes.
Public Cloud cloud have their shit together, but they also deal with millions of hard drives. I could definitely see a story coming out where someone finds a hard drive, some sensitive stuff on it, and just nobody has any idea how it got out. Stranger things than that happen every day.
If it's encrypted then that's another layer of swiss cheese.
It lowers the risk a minimum amount (which makes it not useless, but close to it.) Your resources are limited, so you want to prioritize actions that have good cost:benefit ratio.
Re-encrypting disks is a significant effort (cost), effort that could be spent on something with better benefit. Should you spend a day encrypting a database or should you spend it on looking over publicly exposed S3 buckets? Ideally both, but resources are limited. Doing one action always means you're putting off something else.
Did you see other comments in this thread, for example someone bought a drive online and turned out it still had some backblaze data?
Compliance often has a bunch of useless checkboxing, but in that case it really mattered.
I heard a rumor that some companies had their backups "in the other tower". People won't be making that mistake again.
In some places they have a policy against two key people being on the same plane. It's ridiculous, until it isn't.
Obviously there are priorities. But you can't say "I need to add features, not unit tests, because the company will go under without these features implemented very soon, and therefore unit tests are close to useless".
Part of it, maybe. But the point about it reducing risk by very little is true.
> Did you see other comments in this thread, for example someone bought a drive online and turned out it still had some backblaze data?
Backblaze data is encrypted, or so they claim. Backblaze is also not hosted on AWS. I've also yet to see any evidence of that claim, though I don't dismiss it.
Data is sharded/spread out over multiple disks, you don't have one disk per customer and have all their data there. You'd get fragments of data. If Backblaze was running their servers on specific disks that were not encrypted, not zeroed, and not destroyed, that'll have to stand for them. Backblaze is hosted in a shared data center/colocation, while AWS has their own data centers with their own personnel. Backblaze is a separate company from AWS.
But also.
> The author is not talking about getting someone with physical access.
Right. The author missed this as one of the major attack vectors that this aims to protect against. I don't think leaving out a real justification for it when saying it's snake oil speaks in favour of the author's point.
Did you read the article? He mentioned that right upfront.
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that"
The article mentions it by dismissing it. No, really, this is the major attack vector this aims to protect against.
That and the ability to two-key the system. No encryption, no two-key.
Most well paid engineers at AWS won't have access either. Presumably some minimal set would, but they would likely be pretty senior, and well paid.
But ideally you'd want one set of people with access to the keys, another with access to the data, and a third with physical access, and no overlap. That way you need three people to conspire.
But that's going to be very hard to achieve in practice. But it's not all or nothing. The closer you get to this goal the harder it'll be for someone to not just do it, but do it without triggering a tripwire from the security folks, or at least persist a log entry that if found would get them thrown in prison.
Public cloud companies are not like a small startup where everyone has root.
(sounds like twitter kinda was, according to recent reports)
[1] The trust moves to the CPU vendors instead of the cloud vendors, but if you don't trust CPU vendors then you're going to have a hard time doing anything with computers in the modern world.
(Note confidential containers builds on top of confidential hardware VMs)
I think the real issue is higher-ups think that if data is encrypted that somehow mitigates meaningful exfiltration, but just look at data breaches like Capital One and you'll see that's not the case.
The whole stealing the (correct) hard drive or concerning yourself with a host-level attack should be at the very bottom of the list as far as your threat model goes -- these are not really typical of the surface area of most deployments and you're better off focusing on the surface area such as application security, transport security, having your applications perform the encryption than relying on AWS controls, etc.
The author is saying the probability that it improves your security is so low that any time it takes away from other security related work is actually making your security worse. That’s a different threat model, and it’s probably the better way to approach security, and the author is probably correct - if he actually uses that saved time to improve security in other ways.
I have a question though, what are the laws that require encryption? I can think of HIPPA, SOC2 maybe? What about GDPR?
For instance, it greatly expands the scope of ‘I lost your data/I exposed your data’ notifications in places like California, and for those under many of those other rulesets.
Someone getting access to a repo of encrypted drive images, or someone losing an encrypted drive, doesn’t count. And for reasonableish reasons.
It’s a basic risk mitigation/blast radius limiting move.
For physical/on-prem especially, since old disks tend to ‘wander’ after retirement, and it’s a great idea to always have had them full disk encrypted to reduce the odds someone sensitive gets exposed years down the line.
Beyond the compliance requirements (e.g. CIS), it’s wrong to assume that disks are always perfectly disposed of – simply never having to worry about featuring in one of those “look what I found on eBay!” posts and notifying your customers is worth the basically non-existent cost of basic encryption. It’s something you turn on and basically never think about again - the performance hit disappeared around 2010 and it’s easy to enable globally.
Beyond that, however, there are two key benefits - the author even almost discovers one of them but didn’t think about it enough: encryption means deletion is probably fast - delete the key and you don’t need to monkey around wiping disks and snapshots.
The other, bigger, one is that it can protect against mistakes or compromises. If you share a KMS-encrypted resource accidentally, an attacker won’t have access to the data. An attacker can’t access a resource with a custom KMS policy unless they compromise the specific role which has access or do more invasive things which will trip alerts. Again, not perfect but it protects against common mistakes like the Capitol One firewall breach, which is why all of those standards started recommending it.
That was not my intention and it's unfortunate that's how it sounds. Just like most people don't need armed guards, most people don't need bank level security for their disks either. Encrypting disk encryption is usually not the most effective thing you can do to improve your security, there are usually other actions with better payoff.
AWS zeroes the disks before they're reused, so from your point of view it's instant. As mentioned in the post, deleting a KMS key takes at least 7 days (and up to 30 days, depending on your configuration.) Regarding other point, that's also mentioned in the post as "if your IAM access configuration is bad your KMS access configuration might save you," which is referring to what you mention.
> It’s something you turn on and basically never think about again - the performance hit disappeared around 2010 and it’s easy to enable globally.
This is true if you, or the one you inherit the infrastructure from, configured it correctly from the start. But re-encrypting disks, databases, S3-buckets etc is time consuming and might require downtime. So it's not always easy or cost free (in terms of labor.) I'm not sure what you're referring to with "easy to enable globally", enable what?
Thanks for sharing your perspective!
Yes, the point is that it separates physical and logical concerns. That's why all of the major services have been moving to have it on by default since it means that there's no way for lost hardware to cause a data breach, and while the KMS delays to prevent accidental / ransomware deletion are not instantaneous they're visible and deterministic: things are deleted when you think they are.
> Regarding other point, that's also mentioned in the post as "if your IAM access configuration is bad your KMS access configuration might save you," which is referring to what you mention.
I guess I'm just not seeing how that's “close to useless and potentially harmful” – defense in depth is a useful hardening technique, especially if you have to worry about admins being compromised or demonstrate that sysadmins can't access data without following a standard logged path.
> This is true if you, or the one you inherit the infrastructure from, configured it correctly from the start. But re-encrypting disks, databases, S3-buckets etc is time consuming and might require downtime.
Saying that something is hard to clean up later isn't “close to useless and potentially harmful” — if anything, I'd take the opposite position that if you do the “pointless” compliance work, you're not going to be in that position later when you have a ton of data and busy applications.
> So it's not always easy or cost free (in terms of labor.) I'm not sure what you're referring to with "easy to enable globally", enable what?
Partially I was thinking about https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncry... but the more general point was just that these are easy to set up when you're creating new resources and there isn't a runtime impediment for having done so the way there was decades ago when performance was a reasonable concern.
In general, when the entire industry is saying you should do something or, as is the case for the major cloud providers, turning it on by default, I would stop to ask whether it is in fact unnecessary or them trying to prevent yet another customer from ending up in a situation where the work required to deal with a problem is considerably greater than preventing it would have been.
The "close to useless" is based on it lowering any risk with very little, it's not a big payoff security wise in most cases. "Potentially harmful" refers to the cost of potentially having to re-encrypt data, that is cost/effort that could instead be spent on other security mitigations with better cost:benefit ratio.
You have a limited amount of resources, and with those resources you want to lower the risk as much as possible. I consider re-encrypting data to fare badly in such calculation. It's a high effort low benefit mitigation in most cases. If encryption is done correctly, it might cost little effort and gives little benefit and might therefore be worth it. It's very common to not do it correctly though, and that requires re-encrypting. I have not found an SCP that allows KMS keys but not default encryption keys, which means manual effort is spent on teaching developers and/or build/use tooling.
How many ways are there to do it wrong than to do it right?
This is partially true with respect to EBS (i.e. virtual disks). When you boot up an EC2 instance, the a plaintext data encryption key is loaded into hypervisor memory. As long as the EC2 instance is still running, you can add an additional EBS volume to the instance and then copy data from the disk that has the orphaned key. Even if the underlying KMS key was deleted.
The rest of this stuff feels like FUD me. Yes, IAM has identity-based policies and KMS has resource-based policies. And yes, default service KMS keys are unique per account (why would you expect otherwise?).
Implementing an encryption program takes forethought. This is true if you're hosting your own data, too. I really don't miss manually rotating drives, fighting with LUKS, and then putting drives in a fireproof safe (which was never locked anyway, so I'm not sure if it would've actually protected anything in an actual fire).
I expect it to be unique per account, but I would be happy if it was possible to share it with other accounts so one could make cross account backups (it's good practice to have a separate AWS account for backups.) Currently this requires a KMS key, which means data encrypted with the default key must be re-encrypted and that takes a lot of time and effort.
https://en.wikipedia.org/wiki/Swiss_cheese_model
AKA defense-in-depth
Relying on one control is a recipe for failure, which is why security measures work best when layered.
You don’t trust in one control, you trust that you stack enough controls that one of them works.
Just because something isn't 100% perfect in every scenario doesn't mean it shouldn't be done at all.
But I agree with your point, if you're really worried about data to the point where you don't trust AWS with encryption keys, you should self-manage your keys and manually encrypt/decrypt data without AWS KMS.
As for some of these statements (i.e. loss of key equals loss of data, re-encryption with a new key is expensive to go through an entire disk, etc.), that is exactly the point of encryption and holds true beyond the cloud. It is not meant to be trivial; it is a layer of security that trades convenience and often some level of performance.
That's pretty much the case for any type of encryption.
This blog post is like someone complaining "parameterized queries are so much more of a pain than just using string interpolation to build my queries. I'd never allow SQL injection in my app." Hubris of an inexperienced engineer imho.
This seems unnecessarily hostile. I've worked with cloud infrastructure and security for more than 10 years. If you have experience of unencrypted disks being stolen from AWS I'm very interested to hear about it.
Note, though, that I don't claim one shouldn't encrypt disks, but I consider it to be very low on the list of priorities when it comes to lowering risk. There is almost always risk-lowering actions with better cost:benefit ratio than encrypting disks in the cloud, since that risk is already so low.
If FDE is easy to do, it’s usually worth it to reduce the risk to zero.
Is this really the only vector for data leakage?
> You used the AWS default key to encrypt a database?
Don’t do that? Create a separate account to hold keys. Lock it down like you would your domain DNS or anything else with security implications that will impact your entire company.
Share granular read permissions across accounts as needed to encrypt / decrypt.
Agreed it’s time consuming to initially setup, but it’s a solved problem and can be implemented with your IaC flavor of choice (CloudFormation directly, CDK, Terraform, …).
What you describe is essentially what I currently do. But I've inherited an infrastructure that was not setup that way, and re-encrypting things has been very time consuming.
The company I'm at now use multiple AWS accounts where teams have their own accounts, and it's common for people to forget to use KMS when creating databases or similar. I might have just failed in my search but I couldn't find any way to block default keys via SCPs. If you have any suggestions for that I'm happy to take them!