Are hard drives getting better? | Hacker News Reader

Are hard drives getting better?

backblaze.com

I'm curious what this data would look like collated by drive birth date rather than (or in 3D addition to) age. I wouldn't use that as the "primary" way to look at things, but it could pop some interesting bits. Maybe one of the manufacturers had a shipload of subpar grease? Slightly shittier magnets? Poor quality silicon? There's all kinds of things that could cause a few months of hard drive manufacture to be slightly less reliable…

(Also: "Accumulated power on time, hours:minutes 37451*:12, Manufactured in week 27 of year 2014" — I might want to replace these :D — * pretty sure that overflowed at 16 bit, they were powered on almost continuously & adding 65536 makes it 11.7 years.)

I think it's helpful to put on our statistics hats when looking at data like this... We have some observed values and a number of available covariates, which, perhaps, help explain the observed variability. Some legitimate sources of variation (eg, proximity to cooling in the NFS box, whether the hard drive was dropped as a child, stray cosmic rays) will remain obscured to us - we cannot fully explain all the variation. But when we average over more instances, those unexplainable sources of variation are captured as a residual to the explanations we can make, given the avialable covariates. The averaging acts a kind of low-pass filter over the data, which helps reveal meaningful trends.

Meanwhile, if we slice the data up three ways to hell and back, /all/ we see is unexplainable variation - every point is unique.

This is where PCA is helpful - given our set of covariates, what combination of variables best explain the variation, and how much of the residual remains? If there's a lot of residual, we should look for other covariates. If it's a tiny residual, we don't care, and can work on optimizing the known major axes.

Well said, and made me want to go review my stats text.

I find it more straight forward to just model the failure rate with the variables directly, and look metrics like AUC for out of sample data.

I personally am looking forward to BackBlaze inventing error bars and statistical tests.

Exactly. I used to pore over the Backblaze data but so much of it is in the form of “we got 1,200 drives four months ago and so far none have failed”. That is a relatively small number over a small amount of time.

On top of that it seems like by the time there is a clear winner for reliability, the manufacturer no longer makes that particular model and the newer models are just not a part of the dataset yet. Basically, you can’t just go “Hitachi good, Seagate bad”. You have to look at specific models and there are what? Hundreds? Thousands?

> On top of that it seems like by the time there is a clear winner for reliability, the manufacturer no longer makes that particular model and the newer models are just not a part of the dataset yet.

That's how things work in general. Even if it is the same model, likely parts have changed anyway. For data storage, you can expect all devices to fail, so redundancy and backup plans are key, and once you have that set, reliability is mostly just a input into your cost calculations. (Ideally you do something to mitigate correlated failures from bad manufacturing or bad firmware)

"Actually HGST was better on average than WD"is probably about the only kind of conclusion you can make. As you have noted, looking at specific models doesn't get you anything useful because by the time you have enough data the model is already replaced by a different one - but you can make out trends for manufacturers.

> if we slice the data up three ways to hell and back, /all/ we see is unexplainable variation

It's certainly true that you can go too far, but this is a case where we can know a priori that the mfg date could be causing bias in the numbers they're showing, because the estimated failure rates at 5 years cannot contain data from any drives newer than 2020, whereas failure rates at 1 year can. At a minimum you might want to exclude newer drives from the analysis, e.g. exclude anything after 2020 if you want to draw conclusions about how the failure rate changes up to the 5-year mark.

Right. Does the trouble at year 8 reflect bad manufacturing 8 years ago?

Honestly, at 8 years, I'd be leaning towards dirty power on the user's end. For a company like BackBlaze, I'd assume a data center would have conditioned power. For someone at home running a NAS with the same drive connected straight to mains, they may not receive the same life span for a drive from the same batch. Undervolting when the power dips is gnarly on equipment. It's amazing to me how the use of a UPS is not as ubiquitous at home.

> It's amazing to me how the use of a UPS is not as ubiquitous at home.

Most users don't see enough failures that they can attribute to bad power to justify the cost in their mind. Furthermore, USPes are extremely expensive per unit of energy storage, so the more obviously useful use case (of not having your gaming session interrupted by a power outage) simply isn't there.

UPSes are a PITA. I have frequent enough outages that I use them on all of my desktops, and they need a new battery every couple years, and now I'm reaching the point where the whole thing needs replacement.

When they fail, they turn short dips, which a power supply might have been able to ride through into an instant failure, and they make terrible beeping at the same time. At least the models I have do their test with the protected load, so if you test regularly, it fails by having an unscheduled shutdown, so that's not great either. And there's not many vendors and my vendor is starting to push dumb cloud shit. Ugh.

Sounds like you have some APS model. I had those issues, and switched to Cyberpower. The alarm can be muted and the battery lasts for many years.

A UPS is a must for me. When I lived in the midwest, a lightening strike near me fried all my equipment, including the phones. I now live in Florida and summer outages and dips (brownouts) are frequent.

I've got Cyberpowers actually. The alarm can be muted, but it doesn't stay muted. Especially when the battery (or ups circuitry) is worn out so a power dip turns into infinite beeping. But also if the computer is turned off.

Many years ago I had the same thing happen - actually came in the phone line, fried my modem and everything connected to the motherboard. More recently I had lightning strike a security camera - took out everything connected to the same network switch, plus everything connected to the two network switches one hop away. Also lit up my office with a shower of sparks. Lightning is no joke.

Drives run off the regulated 12V supply, not the raw power line. "Dirty power" should not be a problem.

It would depend on how well done the regulation was in the power supply, wouldn't it?

why people continue to misunderstand this befuddles me. If you bought a budget PSU, then who knows what the voltages really are coming down the +3/+5v lines. You hope they are only +3/+5, but what happens when the power dips. Is the circuitry in the bargain priced PSU going to keep the voltages within tolerance, or do they even have the necessary caps in place to handle the slightest change in mains? we've seen way too meany tear downs to show that's not a reliable thing to bank your gear on.

PSUs don't really rely on caps to maintain voltage, there are negative feedbacks on top of negative feedbacks.

> why people continue to misunderstand this befuddles me.

You might want to check whether your befuddlement is due to your own misunderstanding of the topic. How many switching regulators have you built? We aren't living in fixed AC transformer days anymore, even the shittiest PSU won't behave like you're making it out. The legally required PFC will already prevent it just by itself, before the main 400V DC/DC step-down even gets its hands on the power. And why are you even mentioning 3V/5V? Those rails only exist for compatibility, modern systems run almost entirely off the 12V rails; even SATA power connectors got their 3.3V (it's not 3V btw) pins spec'd away to reserved by now.

I work there. Can't go into much detail, but we have absolutely had various adventures with power and cooling that were entirely out of our control. There was even an "unmooring" event that nearly gave us a collective heart attack, which I'll leave you to guess at :)

> it’s amazing to me how the use of a UPS is not as ubiquitous at home

I live in the UK. I’ve had one power cut in the last… 14 years? Brown outs aren’t a thing. I’ve had more issues with my dog pulling out cables because she got stuck on them (once) than I have with any issues to my supply

Living in Sweden, realized last year that I hadn't replaced my homeserver/NAS in a long time, still haven't had time to replace it and the 2 drives (WD RED) are now approaching 10 years of power on time without any smart problems so far.

Agreed, these type of analyses benefit from grouping by cohort years. Standard practice in analytics.

Over the past couple of years, I've been side hustling a project that requires buying ingredients from multiple vendors. The quantities never work out 1:1, so some ingredients from the first order get used with some from a new order from a different vendor. Each item has its own batch number which when used together for the final product yields a batch number on my end. I logged my batch number with the batch number for each of the ingredients in my product. As a solo person, it is a mountain of work, but nerdy me goes to that effort.

I'd assume that a drive manufacture does similar knowing which batch from which vendor the magnets, grease, or silicon all comes from. You hope you never need to use these records to do any kind of forensic research, but the one time you do need it makes a huge difference. So many people doing similar products that I do look at me with a tilted head while their eyes go wide and glaze over as if I'm speaking an alien language discussing lineage tracking.

Are you using a merkle tree for batch ids?:

  crust = f({flour, butter})
  filling = f({fruit, sugar})
  pie = f({crust, filling})

…where f = hash for a merkle tree with fixed size (but huge!) batch numbers, and f = repr for increasingly large but technically decipherable pie IDs.

No. That sounds like someone that's had some sort of theory before. I just started up an uneducated man's database in the form of a spreadsheet. My batch 1001 sheet has all of the data necessary. My batch 1002 sheet has all of its data. I'm just a simpleton when it comes to this stuff.

> it is a mountain of work, but nerdy me goes to that effort.

Presumably required for compliance, if you're selling your products..

Are there decent softwares for tracking this? Or do you use custom spreadsheets or something?

What I'm aware of in the industry is that the large ones use SAP. Dunno if SAP provides that from scratch or if it's custom-built though, but I'd take a guess that it comes with their standard inventory/materials tracking/invoice handling package.

Edit: looked it up, yep, part of SAP HANA LO-BM [1].

[1] https://help.sap.com/docs/SAP_S4HANA_ON-PREMISE/4eb099dbc8a6...

Every decent ERP has it, even Microsoft ones, because anyone trading in any kind of food (batch tracking) or anything tech (serial nos) need it.

I didn't even know Microsoft had an ERP solution. Wtf

Why wouldn't they? They bought a number of established and emerging solutions and integrated them into their business licensing and sales, upgrading them as their platforms evolved. Some date back 40 years, starting as a standalone DOS program, now a SaaS.

Highly relevant: TIL that Microsoft uses SAP software, despite competing with SAP with its own ERP software (Microsoft Dynamics) <https://www.reddit.com/r/todayilearned/comments/1jo08m3/til_...>

I just listed everything in spreadsheets. I have no insights into what real companies do. It's a major case of fake it till you make it. I know enough to that tracking this info is a good idea, so I brute forced it. I now at least have the data to use in a real system if that ever needs to be. Unlikely. I don't even have a company formed or do any kind of sales. It's all been a very involved hobby, with products given away. It's just been a fun way of being able to push back from the keyboard while not sitting on the couch in front of yet another screen.

Yes this is fairly standard in manufacturing environments. builds of material and lot or down to serial # level are tracked for production of complex goods.

(with a tinfoil hat on) I'm convinced that Backblaze is intentionally withholding and ambiguating data to prevent producing too-easily understood visualization that Seagate is consistently the worst of the last 3 remaining drive manufacturers.

Their online notoriety only started after a flooding in Thailand that contaminated all manufacturing clean room for spindle motors in existence, causing bunch of post-flood ST3000DM001 to fail quickly, which probably incentivized enough people for the Backblaze stat tracking to gain recognition and to continue to this date.

But even if one puts aside such models affected by the same problem, Seagate drives always exhibited shorter real world MTBF. Since it's not in interest of Backblaze or anyone to smear their brand, they must be tweaking data processing to leave out some of those obvious figures.

I don't think so, their posts still have all the details and the Seagates stick out like a very sore thumb in their tables:

https://backblazeprod.wpenginepowered.com/wp-content/uploads...

and graphs:

https://backblazeprod.wpenginepowered.com/wp-content/uploads...

Since it's not in interest of Backblaze or anyone to smear their brand

It is if they want to negotiate pricing; and even in the past, Seagates were usually priced lower than HGST or WD drives. To me, it looks like they just aren't as consistent, as they have some very low failure rate models but also some very high ones; and naturally everyone will be concerned about the latter.

OTOH, Seagate never sold customers SMR drives mislabeled for NAS use.

Ugh, source on that? In the market for a new NAS/Homeserver soonish (realized my drives are almost at 10 years of power on time) and would like to have spinning rust behind ssd for larger storage.

It was a whole thing a while back. This was maybe the original article, but once it landed this was the headline of all tech news for a couple of days. https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shin...

SMR drives aren’t inherently bad, but you must not use them in a NAS. The may work well, up until they don’t, and then they really don’t. WD snuck these into their Red line, the one marketed at NAS users. The end result after a huge reputational hit was to promise to keep the Red Pro line on HMR, but the plain Red line is still a coin flip, AFAIK.

I will not use WD drives in a NAS. It’s all about trust, and they violated it to as astonishing degree.

Ugh, that's sad. The 10 year old drives for me are WD:RED's so can't complain about the old ones.

Looking at the Pro/Plus PDF's it seems that they do specify CMR officially so sneaking in SMR disks sounds like lawsuit material.

I'll probably go for them again, just be vary with research!

Not 100% sure about SMR situation, but granted, Seagate was never not technological front runner nor untrustworthy nor unfaithful company; their 5k4 drives were always more cost effective than anybody and they're the first to ship HAMR drives right now as well. It's __JUST__ that the MTBF was always statistically shorter.

Which is a significant “just”, to be sure! But in my experience, if an Iron Wolf survives a RAID rebuild, it’s probably going to work for many more years. I’ve had 3 WD Reds claim to keep working, and still pass their SMART short and long tests, but tank in performance. I’d see the RAID usage graphs and all drives would be at like 5% IO utilization while the Red was pegged at 100% 24/7. The whole volume would be slow as it waited for the dying-but-lying Red to commit its writes.

In each case, I yanked the Red and saw volume wait times drop back down to the baseline, then swapped in an Iron Wolf. Fool me thrice, shame on all of us. I won’t be fooled a 4th time.

I’m not a Seagate fanboy. There’s an HGST drive in my home NAS that’s been rocking along for several years. There are a number of brands I’d use before settling for WD again. However, I’d make sure WD hadn’t bought them out first.

To Seagate's credit though, their warranty service is excellent. I've had the occasional exos drive die (in very large zfs raids) and they do just ship you one overnight if you email an unhappy smart over. Also their nerd tooling, seachest, is freely downloadable and mostly open source. That's worth quite a lot to me...

(And if anyone is curious about their tools – https://github.com/Seagate/openSeaChest is the link. Lots of low level interesting toys!)

Didn't the ST3000DM001 fail because of a design flaw on the read head landing ramp?

According to Wikipedia: https://en.wikipedia.org/wiki/ST3000DM001

Somewhat tangent: Imagine my dismay after googling why two of my drives in my NAS failed within a couple of days of one another, and I came across a Wikipedia page dedicated to the drive's notoriety. I think this is one of the few drives that were so bad that it had it's own dedicated Wikipedia page.

Same thing happened to me with this drive: Lost the data on a RAID because two drives failed at the same time.

I did not even have a raid, I was too cheap to allocate drives to parity. So I had two of these and a couple of other drives in a JBOD array fused together into a logical filesystem using something I can't remember now.

Maybe my approach was better in hindsight: I only lost the data on those two drives and everything else was left untouched. I guess if you know the drives are dodgy, put them in a JBOD array as it will limit the blast radius in case of failure.

That drive's probably earned a quiet retirement at this point

Reminds me of the capacitor blight of the late aughts. Knowing when the device was manufactured helped troubleshoot likely suspect components during repair.

I have a 13 years old NAS with 4x1TB consumer drives with over 10y head flying hours and 600,000 head unloads. Only 1 drive failed at around 7 years. The remaining 3 are still spinning and pass the long self test. I do manually set the hdparm -B and -S to balance head flying vs unloads, and I keep the NAS in my basement so everything is thermally cool. I'm kinda of hoping the other drives will fail so I can get a new NAS but no such luck yet :-(

I admire the "use it until it dies" lifestyle. My NAS is at 7 years and I have no plans to upgrade anytime soon!

The problem with setting a nearly maintenance free nas is that you tend to forget about it just running away in the background.

Then a drive fails spectacularly.

And that's the story of how I thought I lost all our home movies. Luckily the home movies and pictures were backed up.

No RAID?

If you are only going to have one of the two, choose backups, preferably off-site, even better soft-offline, over RAID.

Of course both is best if you don't consider the cost of doubling up your storage (assuming R1/R10) and having backup services to be a problem.

RAID isn't a backup, it only handles certain/specific failure scenarios.

It's actually in the name: R = Redundant, i.e. availability.

Just because it's in the name, doesn't mean it should be considered a fact or best practice in accordance with reality. I think this[0] reddit post frames it in the simplest way possible: "A backup is a copy of the information that is not attached to the system where the original information is."

There are many[1], many[2], many[3] articles about why "RAID is not a backup". If you google this phrase, many more people who are considerably more intelligent and wise than myself, can tell you why "RAID is not a backup" and it is a mantra that has saved myself, friends, colleagues and strangers alike a lot of pain.

[0] https://www.reddit.com/r/storage/comments/hflzkm/raid_is_not...

[1] https://www.raidisnotabackup.com/

[2] https://serverfault.com/questions/2888/why-is-raid-not-a-bac...

[3] https://www.diskinternals.com/raid-recovery/raid-is-not-back...

edit: formatting

The I used to stand for "inexpensive" too, until RAID drives turned out to be everything but. They've since made it a backronym as "independent", although the drives really aren't independent either.

Yes, it covers exactly the "Then a drive fails spectacularly." case. Unless you were hit by some subtle silent data corruption across the RAID (but it's pretty rare compared to classic drive failure with buzzing and clicking sound).

True, it does cover that specific case.

But it doesn't cover the your RAID controller dying, your house burning down, burglary, tornado, tsunami, earthquake and other "acts of god", etc.

"A backup is a copy of the information that is not attached to the system where the original information is."

[0] https://www.reddit.com/r/storage/comments/hflzkm/raid_is_not...

> But it doesn't cover the your RAID controller dying

One of the reasons some people ditch the hardware RAID controllers and do everything in software. If you're at the point of pulling the drives from a dead enclosure and sticking them in something new it's really nice to not have to worry about hardware differences.

I agree, RAID is not a backup (and nobody said it is, in this thread). But if you self-host a lot of data, even as a hobby, it will make your life easier in case of disk failure.

I thought it was implied with "No RAID?" in response to data loss (wherein they mentioned that they had a backup :)

I'm personally very skeptical as I have been using/used RAID for 20+ years, and I have lost data due to:

- crappy/faulty RAID controllers: who actually spends money to buy a good hardware controller, when a cheap version is included in most Motherboards built in the last 15+ years? In one case (a build for a friend), the onboard controller was writing corrupt data to BOTH drives in a RAID-0, so when we tried to recover, the data on both drives was corrupt.

- Windows 8 beta which nuked my 8-drive partition during install

So is that high usage compared to backblaze?

Is the 10y head flying for each head? Is it for heads actually reading/writing, or just for spinning drives/aloft heads?

I only skimmed the charts, they seemed to just measure time/years, but not necessarily drive use over time.

This NAS, a lenovo ix4-300d, came with Seagate drives (ST1000DM003), so its whatever the SMART 240 counter (Head flying hours) means to Seagate I guess. I just interpret it as "not parked", so it could be doing anything, but this NAS is not doing huge amounts of I/O - mostly just for music, movies, and some personal files. I think all the heads for all platters are on one assembly so they are either all parked or all spinning.

I built my home NAS in 2017 the two original drives were replaced after developing bad blocks (4 and 5 years, respectively). The two expansion drives (2018, 2021) are still fine.

I built a NAS for a client, which currently has 22 drives (growing bit by bit over the years) in it (270 GB of raw capacity) and since 2018 has lost only 3 drives.

I’d have thought 2 new drives to replace all that would be worth the investment in power savings alone.

Once i made a FreeNAS and i lost all the wedding photo's. The significant other was not amused and i vowed to use a lot of backups. I have a lot of old NASes, from NetGear to Qnap to Synology. Perk of the job.

But these days i use a Synology DS2412 in a SHR RAID6 configuration. Only 1 of the 12 drives failed thus far, but maybe this is because most of the time it's powered off and activated using WakeOnLan. For day to day i use an old laptop with 2 SATA 1TB disks in a Debian configuration. Documentation and photo's get frequently backupped to the big nas and the big nas uses Hyperbackup to a Hetzner storage that costs me around $5 a month. So now they're in three systems, two different media and one other different place. It would be a pain to restore when the house burns down, but its doable.

That reminds me.. i should document the restore process somewhere. There is no way the other family members can do this right now.

>and i lost all the wedding photo's

And you didn't have a backup? Ouch. I'm sorry for you.

>i should document the restore process somewhere. There is no way the other family members can do this right now.

I agree. If I passed away, or something seriously bad happened to me, nobody in my family would be able to recover any memories.

I should document how to recover all the data in a simple way. And probably print the document and store it somewhere easily accessible.

modern hdds should not be stored powered down.

they should be spinning most of the time in indle to lubricate things.

or so I've heard.

i have my nas setup as such and have 10y drivers with constant success (they move from main to spare after 5y). i also aim for the 30w amd cpu (which drawn around 5w in idle)

for drivers i spend $300 every 5yr on new ones, so i can keep growing and renewing. and is a pretty low cost considering cloud alternatives.

I get where you're coming from, but in the field can somehow differ greatly from theory. I have had no real errors since i've used this big NAS.

I do admit i personally have a lucky track record regarding harddisks. I think in the more than 25 years i have used spinning hardddisks, about 3 or 4 ever failed me. I don't know why, but most technology i use just keeps on working pretty long. :)

I stil have lots of 500 and 1TB disks around in various old NAS devices i haven't booted up in ages. When electricity got quite expensive i decided to stop using those.

The amount of data i really really want to protect is less than 1TB in total i think. All the other stuff on the big NAS is 'nice to have' but not life crushing should it be gone forever.

I (and surely others) would love to know the reason(s) for the FreeNAS failure i.e. what kind of configuration did you have and what went wrong?

Oh nothing fancy i'm afraid. The WD disk it was on just died. There was no RAID of any other rebuild back then, just a single disk. Luckily we got about 80% of the pictures back via family and friends. Rebuilding that from various sources was a pain though. It taught me to use backups the hard way :)

I'm confused - you used FreeNAS on a single disk? I wasn't even aware that was possible.

Thankfully you were able to recover! I think almost everyone has learned to make backups the hard way at some point. I am the local IT guy among a lot of friends (and by extension, their friends) and so I was always the go-to when things got bad. At some point 15+ years ago, I bought "Restorer2k" which was able to save a lot of data from not-quite-dead drives, some of had to go into the freezer overnight in ziplock bags to try and unlock a frozen read-head (rarely helped), others I was able to replace the controller via eBay. One friend lost an almost finished PhD dissertation (months of work). I remember the tears when I was able to recover that :)

At some point I got tired of data recovery and started telling people to buy a Mac + cheap external drive and use the Time Machine feature. Interestingly, I haven't gotten many phone calls the last few years ;)

Err yeah. There were more disks, but I wasn't that experienced, so i made Volumes/SMB shares per disk. FreeNAS itself was on a bootable USB stick. Who needs a big huge single volume eh? :) This was back in 2007 as well.

Have you powered it down lately? Some of them power down and never come up again.

No the entire thing is on a UPS and uptime routinely will say something like 800 days. I also have a whole-home generator so I'm hoping it stays on forever ;-) I also back it up online through IDrive and take frequent local backups so I don't care if the entire thing fails.

Tha's the most responsible form of sabotage I've ever heard

My 15TB DS1511+ from 2011 would like a word.

I only recently replaced a failed HDD and power supply, but otherwise going mostly strong. It will stop responding to the network out of the blue on occasion, but a power cycle gets it back in order.

But I’ve had redundancy for a while with S3, then later (and currently) BackBlaze.

I’ve been looking into replacing it, but I’m hearing Synology hardware and software isn’t as great as it used to be, which is unfortunate, because this thing has been a tank.

Personal anecdote - I would say (a cautious) yes. Bought 3 WD hard drives (1 external, 2 internal, during different time periods; in the last 10+ years) for personal use and 2 failed exactly after the 5 year warranty period ended (within a month or so). One failed just a few weeks before the warranty period, and so WD had to replace it (and I got a replacement HDD that I could use for another 5 years). That's good engineering! (I also have an old 500GB external Seagate drive that has now lasted 10+ years, and still works perfectly - probably an outlier).

That said, one thing that I do find very attractive in Seagate HDDs now is that they are also offering free data recovery within the warranty period, with some models. Anybody who has lost data (i.e. idiots like me who didn't care about backups) and had to use such services knows how expensive they can be.

> replacement HDD that I could use for another 5 years

But the warranty lasts only 5 years since the purchase of the drive, doesn't it?

Yes, but the warranty is "irrelevant" when the drive actually last the whole 5 years (in other words, I am hoping the replacement drive is as well-engineered as its predecessor and lasts the whole 5 years - and it has so far in the last 3+ years).

> 5 years [...] well-engineered [...]

Programmed obsolescence, evilness, should not be rewarded like this.

That has been my experience (and conclusion) too with WD - it does seem very much like it has been engineered for planned obsolescence. That's why I chose to buy from their competitor this year. Consumers though have less and less option today as WD has already gobbled up the others and now there are only 3 major ones left - Seagate, Toshiba and WD. The other alternative is flash drives, but price and data recovery is a big issue with it that holds me back from fully switching to it.

HDD manufacturers offering data recovery...kind of makes sense, and I'm surprised it's never been offered before. They're in a much better position to recover data than anyone else.

I've bought a lot of WD drives over the years and my experience is they used to last 3 years (back when there was a 3 year warranty) and die right after the warranty expired. I think western digital does a very good job making their drives last to the end of the warranty and not a minute longer.

If hard drives increase in capacity while maintaining the same MTBF, does this count as an improvement? If you previously stored your data on 10 drives and now you can store the same data on 5 drives, that reduces the probability of failure of the system as a whole, right? Is there some kind of "failure rate per byte" measure that normalizes for this?

I don't know about this exact metrics, but the Backblaze hard drive report is always a very good read when thinking about failure rates. Maybe check it out and see if you'll get your answers there.

It depends on what you’re doing and what you’re concerned about.

For a simplified example suppose you have X drives storing 20TB vs 2X drives with 10TB in a simple RAID 1 configuration. When a drive fails there’s a risk period before its contents are replicated on another drive. At constant transfer speeds larger disks double that period per drive but half the number of failures. Net result the risk is identical in both setups.

However, that assumes a constant transfer speeds, faster transfer rates reduce overall risks.

Hmm, I hadn't considered that doubling the drive size doubles the resilver time and therefore doubles the exposure time for risk of array loss. I guess the math gets complicated depending on RAID topology.

> If you previously stored your data on 10 drives and now you can store the same data on 5 drives, that reduces the probability of failure of the system as a whole, right?

Well it also means in the case of failure you get 2x the spread of damage across the same amount of data

When I am projecting prices I tend to assume a 5 year life for a consumer hard drive. I do wonder from this data and the change in purchasing from backblaze if the enterprise class drives might pay for their extra price if they survive out to more like 9 years. 20% extra cost per TB verses about 30%+ more life time. They do tend to consume a bit more power and make more noise as well. I wish they had more data on why the drives were surviving longer, if its purchasing in palettes there isn't a lot we can do, but if its that enterprise drives are a lot better than NAS or basic consumer drives then that we compare cost wise.

Slightly related: Anyone noticed how sloppy Samsung has become with the reliability of their NVMe firmware? I learned a lesson when I nearly lost a ton of data in a ZFS mirror of two Samsung Pro 990 4TB NMVes, they might have been from the same batch and had the same firmware version. First one NVMe abandoned the system, then I made a ZFS send of a snapshot of the remaining one, followed by an rsync to have the raw data directly accessible as well, and during the rsync the second NVMe also abandoned ship. A reboot fixed it, and all data was still there, but only upgrading the firmware (to one which now has been superseded again) fixed the problem. I'm really considering no longer buying Samsung Pro NVMes.

Been hearing of drive failures lately with Samsung, Sandisk, and even Crucial.

Possibly because the bulk of recent drive production is getting reserved by the AI datacenters?

The 990 Pros have a temperature sensor issue. See https://github.com/openzfs/zfs/discussions/14793#discussionc...

I think this was either a `number of TBW (terabytes written)` or `% of space used` issue, since both got removed by the OS within 9 hours delta and the same usage (couple of months, ~500GB), because they were in a mirror since the beginning. If it were a sensor issue, SMART data should have shown this. Not saying that a sensor issue does not exist, but I doubt that this was my problem (mine don't have a preinstalled heatsink, I prefer to use my own).

I now use a 3-way mirror and am mixing brands.

One very nice thing: the Samsung Pro 990 4TB has the exact same space (down to the byte) as the WD_BLACK SN850X 4TB, so they can be replaced without any issues. This rarely was the case with SSDs and HDDs and probably other NVMes. Looks like they learned.

I lost about a month of family chats to a Samsung NVME. Yes, I was taking daily backups of the DB, but the damn drive didn't fail catastrophically and instantly. It just started silently corrupting the database, causing little errors until I noticed a month later. By then, my only option was to restore from my backup right before the first error.

The irony is that I'm a huge BTRFS fan, and use it on all my desktops. But this was a database on a server, so of course use EXT4 and be fine with silent data corruption. :/

Per charts in TFA, it looks like some disks are failing less overall, and failing after a longer period of time.

I'm still not sure how to confidently store decent amounts of (personal) data for over 5 years without

  1- giving to cloud,
  2- burning to M-disk, or
  3- replacing multiple HDD every 5 years on average

All whilst regularly checking for bitrot and not overwriting good files with bad corrupted files.

Who has the easy, self-service, cost-effective solution for basic, durable file storage? Synology? TrueNAS? Debian? UGreen?

(1) and (2) both have their annoyances, so (3) seems "best" still, but seems "too complex" for most? I'd consider myself pretty technical, and I'd say (3) presents real challenges if I don't want it to become a somewhat significant hobby.

Would tapes not be an option?

Not great for easy read access but other than that it might be decent storage.

I looked at tape a little while ago and decided it wasn't gonna work out for me reliability-wise at home without a more controlled environment (especially humidity).

>Would tapes not be an option?

AFAIK someone on reddit did the math and the break-even for tapes is between 50TB to 100TB. Any less and it's cheaper to get a bunch of hard drives.

Unless you're basically a serious data hoarder or otherwise have unusual storage requirements, an 18TB drive (or maybe 2) get you a lot of the way to handling most normal home requirements.

Personally, I buy the drives with the best $/storage ratio. Right now that seems to be ~3-6TB drives. Many PC enclosures and motherboards can fit 8-12 drives, fill it up with the cheapest stuff you're willing to spend money on. It will probably break even or be cheaper than the larger drives.

It depends on the use case. As with CPUs, I tend not to buy the top-end but it may make sense to just buy for expansion over time. I think my RAID-1 Synology drives are 8TB. But mostly just use external enclosures these days anyway. Pretty much don't build PCs any longer.

Tapes would be great for backups - but the tape drive market's all "enterprise-y", and the pricing reflects that. There really isn't any affordable retail consumer option (which is surprising as there definitely is a market for it).

I don't know why you were downvoted, I think for the right purpose tape drives are great.

Used drives from a few generations back work just fine, and are affordable. I have an LTO-5 drive, and new tapes are around $30 where I am. One tape holds 1,5TB uncompressed.

I think they are great for critical data. I have documents and photos on them.

Is there not a problem where companies stop manufacturing the tape media because it is obsolete?

I'm not 100% up to speed with the current standing of things, but tapes (specifically the LTO technology) is still being relied on very heavily by the enterprise, both in data centers for things like cold storage or critical backups, and other corporate uses. Archival use is also very strong with libraries and other such institutions having large tape libraries with autoloaders and all that automation jazz. The LTO-5 generation I mentioned was released in 2010, and the first LTO generation was released in 2000 I believe. The current generation is LTO-10, with an uncompressed capacity of 30TB. New LTO tapes are still being produced, the last batch I purchased was made in 2023.

The LTO consortium consists of HP, IBM and one other company I believe. Now, in my opinion, none of this guarantees the longevity of the medium any more than any other medium, but when I initially looked into it, this was enough to convince me to buy a drive and a couple of tapes.

My reasoning was that with the advertised longevity of 30 years under "ideal archival conditions", if I can get 10 years of mileage from tapes that are just sitting on my non-environmentally-controlled shelf, that means I'll only have to hunt down new tapes 3 times in my remaining lifetime, and after that it will be someone else's problem.

> 2- burning to M-disk, or

You can't buy those anymore. I've tried.

IIRC, the things currently marketed as MDisc are just regular BD-R discs (perhaps made to a higher standard, and maybe with a slower write speed programmed into them, but still regular BD-Rs).

Hard drive failure seems like more of a cost and annoyance problem than a data preservation issue. Even with incredible reliability you still need backups if your house burns down. And if you have a backup system then drive failure matters little.

One method that seems appealing:

1. Use ZFS with raidz

2. Scrub regularly to catch the bitrot

3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.)

4. Bring over pizza or something from time to time.

As to brands: This method is independent of brand or distro.

This works great although I should really do step 4 :)

>3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.)

Unless you're storing terabyte levels of data, surely it's more straightforward and more reliable to store on backblaze or aws glacier? The only advantage of the DIY solution is if you value your time at zero and/or want to "homelab".

I'd much rather just have a backblaze solution and maybe redundant local backups with Time Machine or your local backup of choice (which work fine for terabytes at this point). Maybe create a clone data drive and drop it off with a friend every now and then which should capture most important archive stuff.

A chief advantage of storing backup data across town is that a person can just head over and get it (or ideally, a copy of it) in the unlikely event that it becomes necessary to recover from a local disaster that wasn't handled by raidz and local snapshots.

The time required to set this stuff up is...not very big.

Things like ZFS and Tailscale may sound daunting, but they're very light processes on even the most garbage-tier levels of vaguely-modern PC hardware and are simple to get working.

yep, also I'm scared of AWS holding my data hostage if I ever decide to travel to Iran

I have a simpler approach that I've used at home for about 2 decades now pretty much unchanged.

I have two raid1 pairs - "the old one", and "the new one", plus a third drive the same sizes as "the old pair". The new pair is always larger than the old pair, in the early days it was usually well over twice as big but drive growth rates have slowed since then. About every three years I buy a new "new pair" + third drive, and downgrade the current "new pair" to be the4 "old pair". The old pair is my primary storage, and gets rsynced to a partition that's the same size on the new pair. Te remainder of the new pair is used for data I'm OK with not being backed up (umm, all my BitTorrented Linux isos...) The third drive is on a switched powerpoint and spins up late Sunday night and rsyncs the data copy on the new pair then powers back down for the week.

> 3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.)

Maybe I’m hanging out in the wrong circles, but I would never think it appropriate to make such a proposal to a friend; “hey let me set up a computer in your network, it will run 24/7 on your power and internet and I’ll expect you to make sure it’s always online, also it provides zero value to you. In exchange I’ll give you some unspecified amount of pizza, like a pointy haired boss motivating some new interns”.

> In exchange I’ll give you some unspecified amount of pizza

You mean, in exchange we will have genuine social interactions that you will value much more highly than the electricity bill or the pizza.

Plus you will be able to tease me about my overengineered homelab for the next decade or more.

About the worst I can imagine happening (other than the new-found ability to rockroll someone's TV as a prank) is that said friend might take an interest in how I manage my data and want a hand with setting up a similar thing for themselves.

And that's all fine too. I like my friends quite a lot, and we often help eachother do stuff that is useful: Lending tools or an ear to vent at, helping to fix cars and houses, teaching new things or learning them together, helping with backups -- whatever. We've all got our own needs and abilities. It's all good.

Except... oh man: The electric bill! I forgot about that.

A small computer like what I'm thinking would consume an average of less than 10 Watts without optimization. That's up to nearly $16 per year at the average price of power in the US! I should be more cognizant of the favors I request, lest they cause my friends to go bankrupt.

/s, of course, but power can be a concern if "small" is misinterpreted.

Or find someone else with a similar backup need and then both just agree to have enough space to host remote backups for the other. I would have to increase my ZFS from N to 2N TB, but that would be less work and cheaper than setting up a backup computer for N TB somewhere else.

I had to check for data integrity due to a recent system switch, and was surprised not to find any bitrot after 4y+.

It took ages to compute and verify those hashes between different disks. Certainly an inconvenience.

I am not sure a NAS is really the right solution for smaller data sets. An SSD for quick hashing and a set of N hashed cold storage HDDs - N depends on your appetite for risk - will do.

I've hosted my own data for twenty something years - and bitrot occurs but it is basically caused by two things.

1) Randomness <- this is rare 2) HW-failures <- much more common

So if you catch hw-failures early you can live a long life with very little bitrot... Little =! none so zfs is really great.

Don’t get me wrong: IMHO a ZFS mirror setup sounds very tempting, but its strength lie in active data storage. Due to the rarity of bitrot I would argue it can be replaced with manual file hashing (and replacing, if needed) and used in cold storage mode for months.

What worries me more than bitrot is that consumer disks (with enclosure, SWR) do not give access to SMART values over USB via smartctl. Disk failures are real and have strong impact on available data redundancy.

Data storage activities are an exercise in paranoia management: What is truly critical data, what can be replaced, what are the failure points in my strategy?

I agree about manual file hashing. For data that rarely changes it also has some benefits.

Some time ago, I ended up writing a couple of scripts for managing that kind of checksum files: https://github.com/kalaksi/checksumfile-tools

There's no worse backup system than that which is sufficiently-tedious and complex that it never gets used, except maybe the one that is so poorly documented that it cannot be used.

With ZFS, the hashing happens at every write and the checking happens at every read. It's a built-in. (Sure, it's possible to re-implement the features of ZFS, but why bother? It exists, it works, and it's documented.)

Paranoia? Absolutely. If the disk can't be trusted (as it clearly cannot be -- the only certainty with a hard drive is that it must fail), then how can it be trusted to self-report that it is has issues? ZFS catches problems that the disks (themselves inscrutable black boxes) may or may not ever make mention of.

But even then: Anecdotally, I've got a couple of permanently-USB-connected drives attached to the system I'm writing this on. One is a WD Elements drive that I bought a few years ago, and the other is a rather old, small Intel SSD that I use as scratch space with a boring literally-off-the-shelf-at-best-buy USB-SATA adapter.

And they each report a bevy of stats with smartctl, if a person's paranoia steers them to look that way. SMART seems to work just fine with them.

(Perhaps-amusingly, according to SMART-reported stats, I've stuffed many, many terabytes through those devices. The Intel SSD in particular is at ~95TBW. There's a popular notion that using USB like this sure to bring forth Ghostbusters-level mass hysteria, especially in conjunction with such filesystems as ZFS. But because of ZFS, I can say with reasonable certainty that neither drive has ever produced a single data error. The whole contrivance is therefore verified to work just fine [for now, of course]. I would have a lot less certainty of that status if I were using a more-common filesystem.)

If you mostly care about data integrity, then a plain RAID-1 mirror over three disks is better than RAIDZ. Correlated drive failures are not uncommon, especially if they are from the same batch.

I also would recommend an offline backup, like a USB-connected drive you mostly leave disconnected. If your system is compromised they could encrypt everything and also can probably reach the backup and encrypt that.

Better how?

With RAID 1 (across 3 disks), any two drives can fail without loss of data or availability. That's pretty cool.

With RAIDZ2 (whether across 3 disks or more than 3; it's flexible that way), any two drives can fail without loss of data or availability. At least superficially, that's ~equally cool.

That said: If something more like plain-Jane RAID 1 mirroring is desired, then ZFS can do that instead of RAIDZ (that's what the mirror command is for).

And it can do this while still providing efficient snapshots (accidentally deleted or otherwise ruined a file last week? no problem!), fast transparent data compression, efficient and correct incremental backups, and the whole rest of the gamut of stuff that ZFS just boringly (read: reliably, hands-off) does as built-in functions.

It's pretty good stuff.

All that good stuff works fine with single disks, too. Including redundancy: ZFS can use copies=2 to store multiple (in this case, 2) copies of everything, which can allow for reading good data from single disks that are currently exhibiting bitrot.

This property carriers with the dataset -- not the pool. In this way, a person can have their extra-important data [their personal writings, or system configs from /etc, or whatever probably relatively-small data] stored with extra copies, and their less-important (probably larger) stuff stored with just one copy...all on one single disk, and without thinking about any lasting decisions like allocating partitions in advance (because ZFS simply doesn't operate using concepts like hard-defined partitions).

I agree that keeping an offline backup is also good because it provides options for some other kinds of disasters -- in particular, deliberate and malicious disasters. I'd like to add that this this single normally-offline disk may as well be using ZFS, if for no other reason than bitrot detection.

It's great to have an offline backup even if it is just a manually-connected USB drive that sits on a shelf.

But we enter a new echelon of bad if that backup is trusted and presumed to be good even when it has suffered unreported bitrot:

Suppose a bad actor trashes a filesystem. A user stews about this for a bit (and maybe reconsiders some life choices, like not becoming an Amish leatherworker), and decides to restore from the single-disk backup that's sitting right there (it might be a few days old or whatever, but they decide it's OK).

And that's sounding pretty good, except: With most filesystems, we have no way to tell if that backup drive is suffering from bitrot. It contains only presumably good data. But that presumed-good data is soon to become the golden sample from which all future backups are made: When that backup has rotten data, then it silently poisons the present system and all future backups of that system.

If that offline disk instead uses ZFS, then the system detects and reports the rot condition automatically upon restoration -- just in the normal course of reading the disk, because that's how ZFS do. This allows the user to make informed decisions that are based on facts instead of blind trust.

With ZFS, nothing is silently poisoned.

Better than RAIDZ1, which is what you suggested, in the sense it's more reliable. I didn't say anything about not using ZFS.

I never mentioned RAIDZ1 (and neither did you, until just now).

It doesn't matter but you can just go look.

If you don't have too much stuff, you could probably do ok with mirroring across N+1 (distributed) disks, where N is enough that you're comfortable. Monitor for failure/pre-failure indicators and replace promptly.

When building up initially, make a point of trying to stagger purchases and service entry dates. After that, chances are failures will be staggered as well, so you naturally get staggered service entry dates. You can likely hit better than 5 year time in service if you run until failure, and don't accumulate much additional storage.

But I just did a 5 year replacement, so I dunno. Not a whole lot of work to replace disks that work.

Get yourself a Xeon powered workstation that supports at least 4 drives. One will be your boot system drive and three or more will be a ZFS mirror. You will use ECC RAM (hence Xeon). I bought a Lenovo workstation like this for $35 on eBay.

ZFS with a three way mirror will be incredibly unlikely to fail. You only need one drive for your data to survive.

Then get a second setup exactly like this for your backup server. I use rsnapshot for that.

For your third copy you can use S3 like a block device, which means you can use an encrypted file system. Use FreeBSD for your base OS.

For am4 it is pretty easy to get things working with ecc udimms. I have a 5900x with 32gb of ddr4 ecc ram ticking in my basement.

I don't understand what you're worried about with 3.

Make a box, hide it in a closet with power, every 3 months look at your drive stats to see if any have a buch of uncorrectable errors. If we estimate half an hour per checkup and one hour per replacement that's under three hours per year to maintain your data.

Offline data storage is a good option for files you don't need to access constantly. A hard drive sitting on a shelf in a good environment (not much humidity, reasonable temperature, not a lot of vibration) will last a very very long time. The same can't be said for SSDs which will lose their stored data in a mater of a year or two.

> The issue isn’t that the bathtub curve is wrong—it’s that it’s incomplete.

Well, yeah. The bathtub curve is a simplified model that is ‘wrong’, but it is also a very useful concept regarding time to failure (with some pretty big and obvious caveats) that you can broadly apply to many manufactured things.

Just like Newtonian physics breaks down when you get closer to the speed of light, the bathtub curve breaks down when you introduce firmware into the mix or create dependencies between units so they can fail together.

I know the article mentions these things, and I hate to be pedantic, but the bathtub curve is still a useful construct and is alive and well. Just use it properly.

I feel like I’d like to see graphs in the shape you see in some medical trials – time on the x axis and % still alive on the y. You could group drives by the year they were purchased and have multiple lines for different years on there.

So I had a random thought about what is the most platters that any hard drive has had. I looked it up it seems that the Western Digital Ultrastar® DC HC690 has eleven platters in a 3.5” form factor. That certainly gives you a lot more bandwidth, though not much help for seek time (unless you do the half-allocated trick).

I wonder why they don't do double height form factors at this point instead of trying to squeeze 11 platters with helium and micron level tolerances

Because most of those drives are getting slotted into sleds in arrays. A double height drive might work fine in a PC case (at least if you're willing to break out a dremel), but the majority of the buyers of big drives are not using ordinary PC cases.

Most of the bandwidth of a drive is out at the edge of the platter, so you don’t want to make them smaller in order to improve the seek time or make the read arms thinner to stack more platters. Which increases capacity less dramatically each time anyway.

There’s an old trick though that if you want a faster drive, only partition part of the space on each drive and thus reduce the average seek time. What one could do is make the axle bigger, shorten the arms, make them a little thinner and add another platter. But the next one is only 9% and that’s if you don’t take 9% off the area to accomplish it. Otherwise you get the same space but faster seek time.

Of note, assuming that decommissioning of drives is driven primarily by e.g. space concerns rather than signs of impending individual drive failures (which seems to be the case based on the linked article about storage scaling), you could conduct a survival analysis in which decommissioned drives are treated as right-censored to get a better measure of the failure rate over time as well as how that failure rate depends on various factors. Note that the most common choice of a proportional hazards model may not be appropriate here, and an accelerated failure time model may be more appropriate, although I couldn't say for sure without actually working with the data.

So hard drives are getting better but just as importantly, we're also getting smarter about how we use and maintain them

Not from the prices I'm seeing.

Recent and related:

Disk Prices https://news.ycombinator.com/item?id=45587280 - 1 day ago, 67 comments

Is backblaze single highhandedly driving QC on hard drive manufacturers with their yearly report?

Might be.

It seems odd to look at failure rate in isolation, without considering cost and density; at scale, improved cost and density can be converted to lower failure rates via more aggressive RAID redundancy, no?

Ah I haven’t seen the yearly backblaze post in some time now, glad it’s back.

Connected, but quite different to this subject, is how to long term store photos (cloud does not count). HDD still seem to be the best solution, but not sure how often should I rewrite them

Print out the ones you like and put them in an album or on the wall. Think how many photos you have that are like that in a family and still around, when all the rest are gone on dead phones or computers somewhere.

You can't get a perfect digital copy of a printed out photo. You're subjecting yourself to generational losses for no good reason.

If you're a fan of paper, you could base64 encode the digital photo and print that out onto paper with a small font, or store the digital data in several QR codes. You can include a small preview too. But a couple hard drives or microSD cards will hold many millions of times as many photos in less physical space.

Doing paper backups is hard I could not store much data on paper when I last tried with QR codes. You need to to multiple QR codes since they are only 2KB each. There are some threads about it on HN, but I have never seen anyone do megabytes per paper

https://news.ycombinator.com/item?id=29792556 https://news.ycombinator.com/item?id=31149427 https://news.ycombinator.com/item?id=24669425 https://hn.algolia.com/?q=paper+backup Cuneiform tables do it in clay.

PaperBack claims 0.5MB per page: https://news.ycombinator.com/item?id=39883531

It's likely not possible to do much better. I never said that was the most practical method.

Photo prints will yellow over time, a lot. You have to choose paper and storage environment carefully if you want them to last.

Hard drives are probably better than paper as long as the power grid is powered and there are still computers with the right kinds of ports to read them.

M-DISC. It's more expensive by size but for (private I'm assuming) pictures it doesn't make a difference.

TBH i've never heard about it before. And I even have a drive that records it! Thank you!!!!

However what I've been thinking before you wrote about M-DISC - was making a set of HDD that I'd dump my photos and videos and rewrite each year for example, i.e. copy from one drive to another. This copying 4TB of data should be enough to store them for a few years.

I'm mostly concerned with family photos and videos, and maybe music, but I tend to buy CDs of the most important music for me. I'd say that other data would be expendible...

You’ll want to protect yourself against bit rot too. Don’t go copying bad files every year.

bit rot... What about good filesystems? Like ZFS which has checksums. Or maybe a zip with additional recovery data... I wonder if there is a ready-made solution that allows this to not require me scripting (and losing those scripts later or failing to run them because Bash was upgraded from 4 to 5 and Python from 3 to 4)

At least if you don't encrypt and those are jpeg it doesn't matter. I lost two discs at once with theWD debacle a few years back and couldn't recover anything as I encrypted the disk. Another drive that got bad, I easily recovered 90% of the files using foremost.

So no encryption on the local backup for me, only the emails dump by encrypting the zip that contains them. It's not perfect but that's the compromise I (think I) have to make. (The remote one is encrypted though)

Could also add par files for additional layer of safety. Plus ofc use zfs or similar with checksums

Hard drives are not getting better.

Hard drives you can conveniently buy as a consumer - yes. There's a difference.

Does this take into account the scandal of old drives being sold as new?

pleasant contradiction to betteridge's law

Do we have enough rare earth metals to provide storage for the AI boom?

The question is, do we have enough capacity to mine and refine them at a reasonable price? They're there, in the dirt for the taking.

Future generations will blame us for damning them out of rare earths to build yet another cellphone. This is like us today with severely diminished whale populations just so Victorians could read the bible for another 2 hours a night. Was it worth it? Most would say no, save for the people who made a fortune off of it I'm sure.

That makes no sense whatsoever. We are not consuming rare earths; only moving them from one place to another.

Arguably, future generations would find it easier to mine them from former landfill sites, where they would be present in concentrated form, than from some distant mine in the middle of nowhere.

Sounds mighty expensive if not impossible for extraction.

Why not do that now?

It's called recycling.

As long as the "virgin" sources are super cheap its not worth it, but the market can change.

Will the NIMBYs of the future allow for rare earths processing in long defunct landfills that in the future are surrounded by residential development? These urban landfills even today really aren't far from civilization. In fact they have been enveloped by civilization in many cases.

I mean as it is you can't even recycle most things if they are the least bit soiled since it is not economically viable to implement a cleaning process. We are doing a whole lot of assumptions that our future members of our species will have solved a way to reliably get pure rare earths from a mixed up slop of everything in a landfill. Whatever they possibly figure out is going to probably be far more challenging than ore refining processes we use today.

It might be cheaper/easier to try and capture an asteroid than to refine a landfill.

> Future generations will blame us for damning them out of rare earths to build yet another cellphone.

We’ll be out of many elements before we run out of rare earths. They are not actually that rare, they are mostly inconvenient to extract because they are distributed everywhere as minor elements rather than concentrated into ores. Things like cobalt, nickel, the platinum group metals, or even copper, are more worrying from a sustainable production point of view.

We used to have several companies in the US that mined and refined the materials. They shut down because they couldn't compete with China, but if supply became constrained over a long period of time they could restart operations.

It's one reason the Chinese threats of cutting off rare earths is not quite as scary as the media makes it out to be. They can't do it for too long before alternatives get spun up and they lose their leverage entirely.

As far as I know most rare earth metals are in fact not that rare. But refinement is concentrated in China.

Reply on news.ycombinator.com