Hacker News Reader

Comment by cm2187 | Hacker News Reader

> Additionally because there are no people, we can take all the oxygen and most of the water vapour out of the atmosphere which reduces corrosion, which is a significant problem in data centres

First time I hear corrosion is a problem in datacentres. I never noticed any corrosion on any machine at home or work. Does it have to do with the cooling of datacentres?

View on HN

I think part of it is that even problems that are small/unnoticeable at the individual level become big when multiplied by a huge N.

For example, I don't think I've ever had a hard drive fail in ~30 years in computing, well without being dropped that is, yet look at the Backblaze reports:

https://www.backblaze.com/blog/hard-drive-benchmark-stats-20...

I have. Including things like the heads falling off and spinning around, grinding everything to junk.

That sickening click sound that a disk drive makes when it's on the verge of failure is ingrained in my head.

I tried putting one of those in the freezer once, and maybe it was a folk remedy, but it worked enough to boot and get my data off!

That must be one of the silliest hacks I've ever heard, right up there with oven-baking electronics to fix loose solder joints. I love it!

It works because of how hard drives work - the heads are extended by slightly heating the bar the heads are connected to. If you're head crashing, you need a little less length, hence freezer.

This isn't quite true, the heads are kept from crashing by air pressure and a crash does irreversible damage. The "clicking" sound is either the heads flicking back and forth as the drive tries to seek, or the central motor trying and failing to start platters spinning which are jammed on the bearing.

What freezing the drive may achieve is either un-sticking it from the bearings (see also the "bang drive on edge" technique), or lowering the thermal noise floor in the electronics enough for marginal components to make it through the boot sequence.

Ah, interesting! Thanks for the correction. I was off the project before we put in the drives obviously. ;)

When I worked for a small computer place years and years ago we'd sometimes "tap" a drive as it was spinning up if freezing it didn't work. Tapping it with a screw driver handle rarely worked, but it was always a last ditch effort to save having to send a drive out -- which most people wouldn't pay for anyway.

I did that with a drive in my computer fairly recently. Although I didn't so much tap it as shake it on startup. It did finally start spinning enough to insure that I had the last few bits of data transferred to a new drive, though. :)

While they may make legitimate points in the article, it's hard to consider someone that makes money on HDD recovery like the authors of the article having an unbiased viewpoint here.

Not OP, but I have had the freezer trick work more times than not. It's a last case type of deal, but when there's nothing to lose try it. Just make sure the drive is sealed up nicely.

Worked in HS for a local computer repair shop. One trick to improve success with the freezer method is to place the drive in a sealed plastic bag with some desiccant packs. Placing a bare drive in the freezer can kill it due to condensation. Leave the drive in the bag at least a few hours to give the desiccant time to work. Like others have said, this is a last ditch effort, and if you really care enough, don't do it if you are willing to send it to a specialist, since it can ruin a recoverable drive. But it does seem to work ~30% of the time (at least w/ deathstars) and I did have success on a clicking Seagate drive last year.

As a teen I managed to 'rescue' 1000's of £/$'s worth of graphics cards from eBay listed as spares/repair with this method. I recall this working about half the time; then again at least 3/10 cards from eBay listed as broken worked immediately when plugged in.

Did you have any "signals" you looked for on the listings to indicate they might just need an oven reflow, or did you just buy everything cheap and hope it worked?

Reminds me of the old Xbox 360 "red ring of death" fix where you could just wrap the box in towels and leave it running for an hour to overheat it, then reboot and it would work again.

Very recently I was able to get a bootlooping N5X to temporarily boot, long enough to recover pictures, by putting it in to the freezer for a couple of hours. Some folks have (claimed to have) had long term success from freezing the bootlooping devices.

There was a type of Seagate drive long ago (I believe mine was a 2GB) that would stop working, apparently because the spindle got stuck. But if you shook it in a rotational manner, it would unstick! The first time I sent it for warranty service and of course they deleted all my data. But the next time it happened I learned about the trick.

I can confirm the freezer trick worked too. This was back before multi-platter high density drives though. I tried it recently with a failed 6TB drives from a friend, and it didn't work.

Yeah this was many years ago. I think the one I did it with was a 250GB from the mid to late 2000's

You may be interested in this interesting archive of hdd-failing-sounds. Note: as it's about 10 years old, it requires flash. http://datacent.com/hard_drive_sounds.php

It definitely works. I just did it a couple of years ago.

In fact, no matter how long I left it in the freezer, I couldn't get all the data off. I ended up freezing it in the freezer, then putting it in a small cooler with a couple ice packs, with the wires hanging out between the cooler and lid. That gave me just enough to grab all the data (well, really, a couple of not-so-small VHDX files).

It was much fun, like a science experiment in high school.

Well, you're exactly part of a huge N of people that appear on the internet, who combined over the years owned an even larger number of drives.

I've seen a whole batch of hard drives fail: one-by-one the company's employees' PCs failed over a period of a week or two. They had all been bought together in a single upgrade.

I (on a Mac) was unaffected (and no doubt a little smug).

My N is just 10, yet I've had two failures :(

I killed a hard drive when I swapped motherboards about 15 years ago. I suspect static electricity.

That's just unbelievable. To even survive the Deskstar era blows my mind. Not a single drive?

That's not all that surprising to me, I've had supremely good fortune with drives. I have a Maxtor Fireball that still "works" (though its SMART has failed.) My current workstation still has drives in it from the three prior workstations; spanning back 7 or 8 years now[1], humming along well outside their warrantied operation. The first disk failure(s) I witnessed came while I worked at a computer repair shop, which would be "a huge N" compared to the number of PCs I've personally owned. My first disk failure on my own personal equipment didn't happen until I built out a home storage array - again a rather large multiplier of number of disks compared to a traditional PC. Even then that failure was because a disk controller got fried, the drive was mechanically sound and the data was completely recoverable.

Admittedly my own experience is probably a bit skewed since I don't have a personal laptop. By the time I got a job that provided me w/ a laptop solid state drives were already quite mature. In my experience doing repair work: laptops were a huge source of our failed drive & data recovery operations.

[1]: I should note these drives all operate nearly 24/7, no five-nines on my personal equipment though ;-P

Up until a couple years ago I could have said the same. I even have a couple deathstars on the shelf that were operational when retired.

In the last couple years I've had several HDDs fail starting with a 2TB Seagate (one of the "stars" in the Backblaze stats.) I've had others fail since, mostly multi-TB drives. I had one nearly ten year old 200GB drive begin to report remapped sectors. I've even had an SSD develop an unrecoverable error.

My dad managed to kill a 2 year old Samsung SSD; it wouldn't even show up on the bios. Tech support was so incredulous they actually had an Electrical Engineer on their staff call him.

I didn't believe him either. I was like "Dad, SSDs are designed to fail in a way you can recover data. What SATA cable are you using? Is the port dead?"

Turns out it was literally the drive.

Much like a poster above, I've got a walking graveyard of previous platter drives in my workstation from various old workstations and they're all fine, but (and I think mine was a Samsung too), my first experiment with an SSD ended in tears. It was fine for similarly about two years, then I had some weird troubles booting about 3 or 4 times, then it just disappeared completely.

I've been very wary of SSDs since then.

I recall hearing of similar failures early on. I don't recall the brand. Apparently the controller goes bad and it just stops responding to SATA commands.

Intel had a strategy that caused them to go into read only mode once they reached their rated lifetime write capacity. All data remained available until they were power cycled whereupon they would intentionally brick themselves.

In my case it is was a Crucial M4. It still tries to operate but will fail the long SMART self test.

Argh trigger alert, I bought four of those in a raid 10.

They even had the audacity to reject my warranty claim.

They sent me back refurbs that didn't work, I just gave up.

My guess is that you change disks often. I've had multiple fails over the years, either from overheating or more often from overuse for extended periods of time (aka more than five years). Although I've never had a failure before the guarantee expired.

Yesterday I lost 3TB Toshiba TD01* series drive 23,75 months in (home) use. Only 4800 hours powered on. And it started to seek fail, even SMART showed it. It failed one week before warranty ended. Ha!

Even managing just a few hundred nodes, I see hard drive failures pretty routinely. Servers hammering on storage devices 24x7 are much different from desktop computers. I've never had a hard drive fail on my own desktop machine, though I know people who have.

The only hard drive I've broken was because a let a pretty powerful magnet fall on the computer, right on the spot the HDD is. Never play with magnets next to your computer.

That backblaze report is absolutely fascinating! I'm curious though what the statistics would be from a more consumer point of view, with regards to quantity of use, operational temperature and so forth. I imagine most consumer drives sit idle almost all of the time, whereas theirs are more likely in constant use.

That said, that paints a pretty handy real world picture of what one might expect.

A trend in energy-efficient data centers is to maintain temperatures and humidity at near the maximum acceptable levels to reduce energy use for cooling and dehumidification.

Data centers used to be like walk-in refrigerators. Now the air is borderline uncomfortably warm and slightly "heavy" with humidity.

Sounds like underground in the desert would be suitable?

Excatly. I know one which only does free-cooling (they only have big fans; that's it).

Failure rate is a bit higher, but they are confident that their architecture is HA/redundant and of what they told me, it's cheaper to replace hardware (they got some DELLs "fresh-air" servers) than to cool the DC.

The industrial scale cooling used in datacentres can produce weird results, like when it rained inside Facebook's first datacentre: https://www.theregister.co.uk/2013/06/08/facebook_cloud_vers...

Sounds like tehy didnt know about this building [1]. That happened like 50ish years ago

1: https://roadtrippers.com/stories/boeing-factory

I wondered about this as well : https://www.researchgate.net/profile/Michael_Schappert/publi...

Seems like its a combination of cooling data centres (flowing air) and the quality of the air being circulated. So if there are pollutants in the air they pose a danger to the circuitry.

Never occurred to me this was yet another thing to be wary of when running a data centre!

I had a PC I built and moved to a bungalow near the beach. Within the year it's power supply had rusted. My last PC in a relatively arid environment lasted 4 years.

It's probably the "sea" environment that would pose corrosion issues.

Power supplies are usually plated or coated recycled "chinesium" steel. This is because it's incredibly cheap and has a half decent EMC outcome for the price. One little nick in the coating or plating and that's it though. Really they could use other materials such as cast aluminium, brass or copper but they are orders of magnitude more expensive.

If you look at some of the older electronic test equipment which was made with much higher standard materials it's not uncommon to find something that has been in a damp shed for three decades and powers up just fine after the dead spiders have been removed and the mould cleaned off. BUT at the time of manufacture they cost more than a mid-range car.

In an interview on Radio Scotland, the project lead said the container is sealed with an inert gas replacing the air and with a significantly lower water vapour content than the atmosphere.

There is a stark difference depending on which sea you are located at. I used to live close to the Atlantic ocean, near the equator. The failure rate of electronic components was through the roof. Random memory failures (due to contaminants in memory contacts) as well as corrosion in copper traces in the PCBs. At least one instance of motherboard fire.

I'm now close to the Pacific. No issues whatsoever.

Yes. The air quality -- especially the amount of sulfur in the air -- contributes to sulfur creep corrosion. Run that through an image search to see the results. It's pretty dramatic. Some of this equipment looks like it had been underwater.

But while the older equipment was pretty stable (so long as the capacitors didn't bulge), today's equipment usually has to comply with ROHS. Lead, for all its faults, has a very well-understood corrosion mechanic. The exotic blends used to replace it, we're discovering, aren't always so great...

You know, now that I think about it, I wonder how many cell phone warranty claims have been denied for water ingress were actually due to poor solder choices? Color-changing stickers aside, of course.

Here's an interesting piece:

http://computer.expressbpd.com/news/ctrls-shows-how-data-cen...

tldr: gaseous contaminants corrode silver/copper components of circuitry

I think it's mainly a problem in highly polluted urban areas because of sulphur dioxide in the air. Most datacenters use enough air conditioning for moisture to be a non-issue.

Some AWS techs in Oregon from a few years ago have a fun story for you.

Maybe it is related to mitigation of sea water facilitated corrosion?

I think it has to do with the humidity of the air.

Reply on news.ycombinator.com