Hacker News Reader

The problem with this is that erase-to-program delay is a major factor in bad write quality.....you really don't want to erase a block and wait a long time before programming it. Where "long" is subjective and there are a lot of details here but the general rule is that the longer you wait after an erase to write, the less accurate your write is (colloquially, your zero decays as it sits)......so in practice while you do erase-ahead, you don't erase too far ahead of your writing and you certainly don't erase at TRIM time in most cases.

Didn’t know that. Don’t suppose you’ve got any links that describe the physical process that results in the correlation between write quality and erase-to-program delay?

How hard would it be to write custom firmware for an M.2 SSD?

I'm not even sure where to start

A good one? Extremely hard. Pretty sure 90% of and SSD firmware is just secret sauce magic to deal with the mess and noise the physical world creates when you try to cram so much entropy into such a tiny physical space.

Do you want research or do you want an analogy? I can give you both. On the research front:

    "An Experimental Analysis of Erase-to-Program Interference in Multi-Level Cell NAND Flash Memories" by F. X. Zhang, et al., in IEEE Transactions on Electron Devices, vol. 63, no. 4, pp. 1621-1628, April 2016.

    "Erase-to-Program Disturbance in NAND Flash Memory: Characterization, Modeling, and Mitigation Techniques" by M. H. Kim, et al., in IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 25, no. 9, pp. 2381-2392, Sept. 2017.

    "A Study of Erase to Program Disturbance in 3D NAND Flash Memory" by T. Wang, et al., in IEEE Transactions on Electron Devices, vol. 64, no. 7, pp. 3153-3159, July 2017.

    "Characterization and Modeling of the Erase-to-Program Disturbance in Multi-Level Cell NAND Flash Memories" by R. Micheloni, et al., in IEEE Transactions on Electron Devices, vol. 56, no. 11, pp. 2384-2392, Nov. 2009.

Don’t suppose you could still provide the analogy. It’ll probably make it easier for me to understand the research as I only have a basic understanding of the physics at play here.

What do you mean less accurate? Data is binary. It either is or it isn't.

Data is binary when you writing software. There’s absolutely nothing binary about data when you’re in the messy world of electronics, where fundamentally everything is analog.

Reading data from multi-level flash chips, or from spinning rust, has more in common with an FM radio, or mobile phone radio, than it does with anything you’ll see in an IDE. It’s all very complex signal processing, trying to extract small signals from lots of noise, in that world you know what your data probably looks like. You then need checksumming to figure out if it’s correct, and if it isn’t you need to try again, or declare the data lost.

But the "encryption" need not be cryptographic strength for this, so it might be reversible even without the key. Consider an old cypher with a small key for example.

Everyone is using AES now.

But where's the key?

Certainly true, but again, anyone that really cares about data destruction is never going to trust TRIM to destroy their data. At a minimum they’re going to use a mechanism that’s designed to destroy data (Secure Erase), and more likely they’re going to use multiple methods to destroy that data.

If you’re in a situation where you’re thinking that TRIM might be good enough, then you clearly don’t care that much about the data potentially being recovered by sophisticated attacker. You just don’t want it to be trivial to recover that data. In that case, even old cyphers and small keys is probably good enough.

But once again, if your really care about destroying your data, use something’s that’s explicitly designed to destroy data!

like this? https://i.imgur.com/mW0d1IE.gif

> Modern SSD controllers encrypt all data by default.

Contrary story from 2018/9:

> The security researchers explain that they were able to modify the firmware of the drives in a required way, because they could use a debugging interface to bypass the password validation routine in SSD drives. It does require physical access to a (internal or external) SSD. But the researchers were able to decrypt hardware-encrypted data without a password. The researchers write that they will not release any details in the form of a proof of concept (PoC) for exploit.

* https://borncity.com/win/2018/11/06/ssd-vulnerability-breaks...

> Microsoft is aware of reports of vulnerabilities in the hardware encryption of certain self-encrypting drives (SEDs). Customers concerned about this issue should consider using the software only encryption provided by BitLocker Drive Encryption™. On Windows computers with self-encrypting drives, BitLocker Drive Encryption™ manages encryption and will use hardware encryption by default. Administrators who want to force software encryption on computers with self-encrypting drives can accomplish this by deploying a Group Policy to override the default behavior. Windows will consult Group Policy to enforce software encryption only at the time of enabling BitLocker.

* https://msrc.microsoft.com/update-guide/en-us/vulnerability/...

> Although Windows ships with its own full volume encryption tool called BitLocker, SSDs that claimed to offer their own hardware-based encryption were trusted by the tool and left alone.

> Now though, after a recent update to Windows 10, Microsoft will assume that connected SSDs don't actually encrypt anything.

* https://www.techradar.com/news/microsoft-will-now-encrypt-ne...

None of that actually seems to contradict the claim that SSDs encrypt everything by default. You're just pointing out various ways that the encryption keys are not kept secure. If you haven't configured your system to make use of the drive's encryption capabilities in a secure way, the drive can (and all evidence points to does) still pass the data through the encryption for whitening without attempting to use the encryption process to provide security.

Do you have any pointers on where I can read this? It’s kind of the first I’m hearing that there’s encryption happening to side west leveling somehow and it’s not clear from a high level description why that would since west leveling is I thought a factor of write locations whereas encryption would only give you protection against bad bit patterns in what you write (but bit patterns afaik don’t impact wear on an SSD).

https://www.researchgate.net/publication/274369001_Data_secu...

Did you link the right paper? There's nothing in there that talks about how encryption would help wear leveling that I could see.

Not got links to details, but you can think about wear levelling at two levels.

The first and most obvious, is wear levelling between blocks of flash memory. Here the SSD controller tracks how often it erases blocks of memory which are probably tens to hundreds of bytes large, and tries to keep those counts nice and even. You’re probably already aware of that type of wear levelling.

But wear levelling across multiple blocks means that there’s nothing tracking how often each cell within a block of memory is being written to and erased, because that depends on exactly what data is written into the block. So if you’re writing data that has a repeating pattern that’s a multiple of your block length, then specific cells within that block will be written to more often than others. This impacts both the wear of the individual cells, but probably also write quality as well, because have two cells next to each other with vastly different stored values almost certainly adds noise when reading again (nature hates a gradient).

You can’t track wear on a per-cell level, because that would require an extreme amount of memory just to store counters, which could be better used to store data. But what you can to is transform the data your writing into “random” data to remove any patterns and produce even wear. Additionally the randomness will also reduce the variance in stored values between adjacent cells, improving write quality.

So encrypting the data provides a bunch of very useful adjacent benefits for a very small cost. Encryption algos like AES are deliberately designed to very high performance on extremely low power, memory constrained devices, and typically have a pretty low transistor count for dedicated crypto silicon. So using strong AES encryption in an SSD is pretty much a no-brainer. You get improved reliability (because better wear levelling), improved performance (because writes are more successful), and feature you charge extra money for (on device full disk encryption).

I'm not sure I follow. Am I correct in understanding that you're saying that if a block has some pattern, then overwriting that block with the same pattern will result in no extra wear on that block but neighboring blocks might overwrite and therefore would have worse wear. Thus encryption smears it out and ensures that you have consistently worst case wear always and that somehow improves reliability?

I'm assuming the reason this works is because the underlying block that gets written isn't the same one that has the old data and thus the probability of overwriting with the same data is low because the encryption key for a given write is somehow derived from block offset + length to make sure that a duplicate pattern for a given block will still encrypt differently?

If I understood that properly, this feels like a lot of complexity for unclear gain (not seeing how it improves write leveling vs at best downgrading the best case / average case to the worst case) and I haven't found any papers online. Is this mechanism something you actually first hand building or something you remember reading from some tech newspaper? Would love a primary source here.

> Am I correct in understanding that you're saying that if a block has some pattern, then overwriting that block with the same pattern will result in no extra wear on that block

> I'm assuming the reason this works is because the underlying block that gets written isn't the same one that has the old data and thus the probability of overwriting with the same data is low because the encryption key for a given write is somehow derived from block offset

No, this has nothing to do with writing the same data to the same blocks, but rather how you handle a continuous stream of data being written (and eventually overwriting older data) to the drive over very long periods of time. You should assume that physical location of your data bears zero correlation to the address to write it to. The SSD controller will always redirect it to least worn available block.

In SSD it’s reasonable to assume that the flash memory can be erased and written at a more granular level than you would want to track for wear levelling purposes (because extra granularity cost more money). So writing repeated patterns of data to a disk that align with the wear levelling block size will result in some parts of your flash memory being written and erased more often. But this unevenness is too granular for the wear levelling algorithm to track and compensate for. Encryption removes the need to track more granular wear patterns because it ensures that every single bit of your flash memory is going to see very similar write patterns.

Additionally there’s an entirely separate aspect, which is that NAND flash is a physical storage medium, and economics mean manufacturers want to squeeze every last ounce of storage out of their silicon. This means pack flash cells physically close to each other, and storing multiple values in each cell. The consequence is that activity in one cell can cause changes in adjacent cells, if those changes are large enough value of a cell will change corrupting the stored data.

By encrypting the data you can eliminate the possibility of extreme edge cases occurring which have a high risk of data corruption. At better place to observe these types of issues is to look at RAM, and Row Hammer attacks. The basic principles are the same, even if the precise physical interactions are a little different.