Hacker News Reader

Comment by fairfeather | Hacker News Reader

SSD firmware engineer here. I work on enterprise stuff, so ymmv on consumer grade internals.

Generally, the data refresh will all happen in the background when the system is powered (depending on the power state). Performance is probably throttled during those operations, so you just see a slightly slower copy while this is happening behind the scenes.

The unused space decaying is probably not an issue, since the internal filesystem data is typically stored on a more robust area of media (an SLC location) which is less susceptible to data loss over time.

As far as how a user is supposed to manage it, maybe do an fsck every month or something? Using an SSD like that is probably ok most of the time, but might not be super great as a cold storage backup.

View on HN

So say I have a 4TB USB SSD from a few years ago, that's been sitting unpowered in a drawer most of that time. How long would it need to be powered on (ballpark) for the full disk refresh to complete? Assume fully idle.

(As a note: I do have a 4TB USB SSD which did sit in a drawer without being touched for a couple of years. The data was all fine when I plugged it back in. Of course, this was a new drive with very low write cycles and stored climate controlled. Older worn out drive would probably have been an issue.) Just wondering how long I should keep it plugged in if I ever have a situation like that so I can "reset the fade clock" per se.

More certain to just do a full read of the drive to force error correction and updating of any weakening data.

noob question... how do i force a full read?

the most basic solution that will work for every filesystem and every type of block device without even mounting anything, but won't actually check much except device-level checksums:

  sudo pv -X /dev/sda

or even just:

  sudo cat /dev/sda >/dev/null

and it's pretty inefficient if the device doesn't actually have much data, because it also reads (and discards) empty space.

for copy-on-write filesystems that store checksums along with the data, you can request proper integrity checks and also get the nicely formatted report about how well that went.

for btrfs:

  sudo btrfs scrub start -B /

or zfs:

  sudo zpool scrub -a -w

for classic (non-copy-on-write) filesystems that mostly consist of empty space I sometimes do this:

  sudo tar -cf - / | cat >/dev/null

the `cat` and redirection to /dev/null is necessary because GNU tar contains an optimization that doesn't actually read anything when it detects /dev/null as the target.

Just as a note, and I checked that it's not the case with the GNU coreutils: on some systems, cp (and maybe cat) would mmap() the source file. When the output is the devnull driver, no read occurs because of course its write function does nothing... So, using a pipe (or dd) maybe a good idea in all cases (I did not check the current BSDs).

This is the most straightforward reliable option:

> sudo dd if=/dev/sdX of=/dev/null bs=4M status=progress iflag=direct

This.

On macOS / Linux you can use `dd` to "copy" everything from /dev/yourssd to /dev/null. Just be careful not to do it the other way!

https://www.man7.org/linux/man-pages/man1/dd.1.html

I have no idea if forcing a read is good / the right way. I'm just answering how to do it.

/dev/null is "empty" if you read it, its /dev/zero you dont want to dd onto your disk.

>Generally, the data refresh will all happen in the background when the system is powered (depending on the power state).

How does the SSD know when to run the refresh job? AFAIK SSDs don't have an internal clock so it can't tell how long it's been powered off. Moreover does doing a read generate some sort of telemetry to the controller indicating how strong/weak the signal is, thereby informing whether it should refresh? Or does it blindly refresh on some sort of timer?

Pretty much, but it depends a lot on the vendor and how much you spent on the drive. A lot of the assumptions about enterprise SSDs is that they’re powered pretty much all the time, but are left in a low power state when not in use. So, data can still be refreshed on a timer, as long as it happens within the power budget.

There are several layers of data integrity that are increasingly expensive to run. Once the drive tries to read something that requires recovery, it marks that block as requiring a refresh and rewrites it in the background.

https://www.techspot.com/news/60501-samsung-addresses-slow-8...

samsung fix was aggressive scanning and rewriting in the background

> maybe do an fsck every month or something

Isn't that what periodic "scrub" operations are on modern fs like ZFS/BTRFS/BCacheFS?

> the data refresh will all happen in the background when the system is powered

This confused me. If it happens in the background, what's the manual fsck supposed to be for?

So you need to do an fsck? My big question after reading this article (and others like it) is whether it is enough to just power up the device (for how long?), or if each byte actually needs to be read.

The case an average user is worried about is where they have an external SSD that they back stuff up to on a relatively infrequent schedule. In that situation, the question is whether just plugging it and copying some stuff to it is enough to ensure that all the data on the drive is refreshed, or if there's some explicit kind of "maintenance" that needs to be done.

How long does the data refresh take, approx? Let's say I have an external portable SSD that I keep stored data on. Would plugging the drive into my computer and running

  dd if=/dev/sdX of=/dev/null bs=1M status=progress

work to refresh any bad blocks internally?

A full read would do it, but I think the safer recommendation is to just use a small hdd for external storage. Anything else is just dealing with mitigating factors

Thanks! I think you're right about just using an HDD, but for my portable SSD situation, after a full read of all blocks, how long would you leave the drive plugged in for? Does the refresh procedure typically take a while, or would it be completed in roughly the time it would take to read all blocks?

Does the refresh procedure take place at all?

I would hope so.

Ok, so all bits have to be rotated, even when powered on, to not loose their state?

Edit: found this below: "Powering the SSD on isn't enough. You need to read every bit occasionally in order to recharge the cell."

Hm, so does the firmware have a "read bits to refersh them" logic?

Kind of. It's "read and write back" logic, and also "relocate from a flaky block to a less flaky block" logic, and a whole bunch of other things.

NAND flash is freakishly unreliable, and it's up to the controller to keep this fact concealed from the rest of the system.

I had to google what 'ymmv' means. To save other people's time – it's 'your mileage may vary'.

Reply on news.ycombinator.com