1-2 year SSD wear on build boxes has been minimal
lists.dragonflybsd.org
lists.dragonflybsd.org
Yeah, the build machine churns a lot. But that work should be primarily done by the FS cache, by buffers. Yes, it's going to write out those small files, but if DragonflyBSD has any kind of respectable kernel, though should be a solid curve, not lots of bursts.
I would love if my old colleague Przemek from Wikia would talk about the SSD wear on our MySQL servers which had about 100k-200k databases per shard.
We wore the _fuck_ out of some SSDs.
You should replace your HDDs with SSDs, though, for a number of reasons, and take the long view, as kryptistk noted the OP is doing. Really compare the cost of SAS 15k drives and Intel 320s or 530s.
But I think in his place, you can take the words of the inimitable Mr. Bergman:
https://www.youtube.com/watch?v=H7PJ1oeEyGg
http://techreport.com/review/27436/the-ssd-endurance-experim...
It rather depends upon the database, no? A 512GB Samsung 850 Pro would last 29 years with 100GB of writes a day in a horrendous, worst case 3x write amplification scenario. Very, very few databases write more than 100GB of data a day, and most are magnitudes less.
-Matt
Nobody gives a shit about a mere 100GB in physical writes to a storage subsystem in a day. A single consumer 512GB crucial can handle a rate like that for almost 30 years.
Write amplification effects are not to be underestimated, but fortunately there are only a very few situations where it's an issue. And as I said, the situations can be largely mitigated by database writers getting off their butts and fixing their backends to push the complexity to the indexes and away from trying to optimize the linear layout of the database by seek-writing into it.
-Matt
How is that relevant to your wrong claim about database 128-byte writes? You word that as if you're correcting me.
I don't even know what point you're trying to make anymore, but you're trying to buttress your argument with what can best be described as diversions.
Most people write far less than they think. Databases aren't particularly magical, any more than the web server log file that is written to a line at a time. Yes, people "give a shit" about a "mere" 100GB of writes, because the vast majority of real projects, including at major corporations, write far less than that per day. So are we just talking about dick measuring now?
Additionally, the WAL has to be synced to disk every commit (unlike a web server log file), and WAL records can be very small. WAL is of course append-only, so you'd hope that a good SSD with a battery/cap backup would cache the writes and flush on the SSD erase block filling up.
All major databases will only deal in the 8KB increments (or whatever their block size is, whether larger or smaller, but never as small as originally claimed) though. They don't write less. Indeed, it's worth noting that most (every single major one) database systems actually write to a sequential transaction log (which they do not have to checkpoint every n-bytes), and only on commit do they actually then make a strategy for changing those source pages and extents, unrolling it from the log and checkpointing it, which by default includes coalescing and combining strategies. The idea that databases are randomly writing small bits of data all over the place is simply wrong, but is the entire foundation of almost every comment in this thread.
https://technet.microsoft.com/en-us/library/aa337560%28v=sql...
As one example. Oracle, pgsql, mysql, and others do the exact same thing.
They aren't randomly changing an int here and a bool there.
I worked on a financial system where we wrote just absurd amounts of data a day. We ran it on a FusionIO SLC device (with a mirror backup), and churned the data essentially around the clock. After three years the FusionIO little lifespan hadn't even moved.
tldr; people grossly overestimate the "magical" nature of databases.
If you have an 8kb block and you change 128 bytes of it, the 'actual update' is much smaller than 8kb. Sure, you're reading/writing 8kb to disk, but everything outside of that 128 bytes is basically fat for the purposes of that change. As I said in my previous post, that can absolutely be mitigated by writes being combined through the checkpointing process, and one would hope that a decent SSD could cope easily with combining writes to an appending log.
A database can still be writing data all over the place. A heavily indexed table can cause quite varied write patterns, which can result in a lot of different pages getting touched. Fortunately, the reality is that well-designed DBs and SSDs are fairly capable of dealing with this.
So now that we're in an understanding that we're talking about database writes to IO, the other matter is how it writes it. I've built a lot of systems on a lot of databases, and the write amplification has generally been very low. I've been building and running significant databases on SSDs for about 7 years now, and while everyone else is finally starting to realize that they're wasting their time if they aren't, we still see the sort of extreme niche fearmongering that makes other people clutch onto their magnetic storage (and I heard it the entire time. "OMG but don't databases kill flash???!?!?". No, certain volumes of writes and types of writes do. Only metering will tell you if that applies). Yes, some people do very odd things that can kill storage, but that is extremely rare. It almost certainly doesn't apply to the overwhelming percentage of HN readers.
It's actually "ironic" in that database systems were built to avoid random IO because it was glacially slow on spinning rust storage. They do everything possible to avoid it, and in actual, empirical metrics running databases on flash the write amplification has been very low, and hardly mirrors the claims throughout this thread.
The SSD's normal internal scans will detect the bits when they start to get flaky and rewrite the block then. It might do some rough tracking to trigger a scan sooner than it would otherwrise, but it is more a firmware issue and not so much a wearout issue.
-Matt
That's taking the long view. :-)
Paradoxically SSDs from two years ago have better endurance than brand new drives now, and a lot better than SSDs in 5 years (unless something big happens).
In anycase, so as long as the SSD is powered and thus able to rewrite blocks as bits start to go bad (creating some wear in the process but not a whole lot compared to normal operation), the actual data will be retained for significantly longer than 10 years.
Unpowered data retention (where there is nothing there checking for and rewriting blocks whos bits start to go bad) depends on cell wear. I didn't pull up the chip specs but my recollection is that it is 10 years for new cells and 1-2 years for a relatively worn cell. Depends on temperature of course. That means I can pull a SSD and shelve it without too much worry for a relatively long time, something that cannot be said for any hard drive.
CD's corrode from the inside out, primarily due to air trapped in the holes I believe. Even though it is laser burned there is no real care taken in constructing the plastic sides or metal film to keep up air so when it gets vaporized the oxygen sticks around and starts corroding the metal. Or the plastic gets fuzzy from UV exposure or age. Totally different ballgame.
Hard drives have even worse problems if left unpowered. I used to pull backup drives and shelve them, but their life spans are radically lowered if left unpowered on a shelf for 6 months and then replugged into a machine at a later time. I'm sure some HDD expert can say why. So I stopped doing that. Now the backup drives stay powered on 24x7.
Hard drives seem to have a limited life span whether you use them or not. I have tons of old HDDs sitting around, some well over 20 years old in fact. If I plug an old HDD in I can sometimes read the data off before the whole thing turns into a brick. I mostly use them to test disk driver error handling.
SSDs left sitting around can always be reformatted (i.e. just using TRIM to clean out the whole thing then start writing to it fresh). Their wear limit is based on rewrite cycles rather than how long they've been sitting around. HDDs are physically destroyed, you don't get a fresh start by reformatting an old HDD.
-Matt
That's kind of mysterious. All things being equal, they should simply experience less wear (no wear, really, other than oxidation and corrosion) than the same drive which has been left plugged in.
I would be happy to be instructed here, though.
RAID controllers fail left and right. we keep tons of spares around.
ssd's fail few and far between, cpus basically do not fail, and memory can go bad but it's exceedingly rare and easy to fix. psu's fail but are easy to fix in modern computers as well (slide-out, redundant, etc.)
having said all that, heat is the primary killer of hardware. if you run a lot of equipment in a dense environment, get a laser thermometer and find your hot spots and fix them with some industrial fans or move your gear around. once your stuff gets hot anything can fail in weird and mysterious ways.
Network cards are subject to lots of signal phenomena that are rare inside the chassis. Long cables are pretty good antennas for certain types of RF signals, so there are all kinds of electrical noises, induced power spikes and other miscellaneous garbage that the network card has to tolerate. Well-shielded cables can help protect the card, but it's definitely one interface that's subject to a bit more electrical abuse than the rest.
Components that have been stressed beyond their tolerances a few times can result in things like signal filters having a lower noise threshold, which makes it harder for the card to pick out the signal from the noise, which leads to more packet loss. After enough abuse, the threshold drops below the usable level and communications halt.
There are lots of factors involved, such as shielding, proximity to nearby radiators, bend radius in cables, cable length, temperature, etc, etc. Whenever I delve into this world, I'm often amazed that anything works at all.
All ways they can. I've found them with blown transistors, dead rats attached, no physical imperfections, etc.
Usually for me it's been some kind of hard failure, eg completely dead.
this mostly happens with the on-board controllers. nics don't fail as often, but we do use high end nics (intel 10g and 4x 1g)
Kind of scary. I would guess the replacement should be perfectly identical, to the last firmware bit (... and giving thanks that no subtle circuit timing factors are involved).
FWIW, lots of hardware from ~30 years ago still works. I have a 27 year old Amiga500 that still boots fine (many of the floppy disks have become unreadable, though).
You can buy fully working vintage computers much older than that on eBay.
30 years would be exceptional if the machine spent its life at the 5-year-target load.
There are two things that benefit from the turnover of machines, one is that Intel stays profitable, the other is that hardware standards have a chance to evolve. The move from ISA->PCI->VESA->AGP->PCIe on video cards would not have been possible had people been holding on to their machines for 3 - 5 years before buying new ones.
On the storage side, I can't help but wonder if it's partially due to incumbent vendors pushing flash as a high markup premium product. I suspect that I am not the only one who has thought, "Well, why replace this array with more already-obsolete magnetic disk that won't be any faster? Wait a year or three, flash markups will come down. Maintenance renewal is cheap."
Now, I realize that cheap flash is dangerous to incumbent pricing structures, so $100K for an all flash tray makes sense when it can replace 10 trays of $30K disk. So maybe they've seen no other choice, business model wise. Maybe it even makes sense to milk a once-per-generation technology disruption for all it's worth for a couple of years before commoditizing it.
But it has kept me happy with what I've already got, since the pricing of the alternative was, for my use cases, hilarious. As opposed to, say, this is somewhat more expensive but we can justify it.
That's one way to look at it. Another is that we might have gone ISA->PCI->PCIe in a shorter overall timeframe b/c there were no distractions to get short-term stuff to market.
The obvious principle here is that innovation happens more rapidly in a market where their his a large demand for improvement and a low friction for upgrades. Pull back on either of those and it slows down the rate of innovation.
When faced with the choice between a Fiesta and a Ferrari, there are many reasons to choose one over the other, but it is ridiculous to say "I picked the Fiesta because the Ferrari has a known tire problem when going 200+ MPH".
The original Intel 40G SSDs could handle an average of 10000 erase cycles (for each block of course), giving the 40G SSD around a 200TB minimum life if you divide it out and then divide by 2 for safety. (Intel spec'd 20TB or 40TB or something like that, 400TB @ 10000 erase cycles, divide by 2 gives you ~200TB or so).
A modern 512G Crucial SSD sits somewhere around a 2000 erase cycle durability, or around 512TB of relatively reliable wear (1PB / 2 for safety).
I would not necessarily trust an SSD all the way to the point where it says it is completely worn out, I would likely replace it well before that point or when the Hardware_ECC_Recovered counter starts to look scary. But I would certainly trust it at least through the 50% mark on the wear status. Remember that shelf life degrades as wear increases. I don't know what that curve looks like but we can at least assume somewhere in the ballpark of ~10 years new, and ~5 years at 50% wear. Below ~5 years and I would start to worry (but that's still better than a shelved HDD which can go bad in 6 months and is unlikely to last more than a year or two shelved).
-Matt
(Some online sources mention "Wear_Leveling_Count", and even misreport that as a percentage – but in fact that seems to be an absolute count of the number of times each single block has been rewritten. The percentage is presumably this Wear_Leveling_Count divided by the rated number of cycles.)
The vast majority of information stored these days is write-once-read-never, followed by write-once-read-occasionally. I expected our developer box which is chock full of uncompressed crash dumps and many, many copies of the source tree in various states to have more wear on it than it did, but after thinking about it a bit I realized that most of that data was write-once-read-hardly-at-all.
In terms of hardware life, for servers there are only a few things which might cut short a computer's life, otherwise it would easily last 50 years. (1) Electrolytic capacitors. (2) Any spinning media or low frequency transformers. (3) On-mobo flash or e2 that cannot be updated.
(1) Electrolytic capacitors have largely disappeared from motherboards in favor of solid caps which, if not over-volted, should last 30 years. Electrolytic caps are not sealed well and evaporate over time, as well as slowly burn holes in the insulator. They generally go bad 10-30 years later depending on how much you undervolt them (the more you undervolt, the longer they last). Even so I still have boards with 30+ year old electrolytics in them that work.
(2) Spinning media obviously has a limited life. That's what we are getting rid of now :-). Low frequency transformers have mostly gone away. Transformers in general... anything with windings that is, have a limited life due to the wire insulation breaking down over time but most modern use cases in a computer (if there are any at all) likely have huge errors of margin.
(3) Firmware stored in E2 and flash, or OTP eprom, rather than fuse-based proms, will become corrupt over time. 10 years is a minimum life, 20-30 years is more common. It depends on a number of factors.
Other than that there isn't much left that can go bad. All motherboards these days have micro coatings which effectively seal even the chip leads, so corrosion isn't as big a factor as it was 20 years ago. The actual chip logic basically will not fail and since the high-speed clocks on the whole mobo can be controlled, so aging effects which degrade junction performance for most of the chip can be mitigated. I suppose an ethernet port might go bad if it gets hit by lightning but I've never had a mobo ethernet go bad in my life. Switch ethernet ports going bad is usually just due to poor parts selection or overvolting which would not be present in a colocation facility or machine room.
In anycase, there is no reason under the sun that a modern computer with a SSD wouldn't last 30 years with only fan, real-time clock battery, and PSU replacements.
-Matt
1. That waxy thermal pad material certainly dries out and becomes ineffective in less than 10 years.
2. Power supplies still have large electrolytic capacitors right next to heatsinks
3. BGA parts with lead-free solder are still going to have a limited amount of thermal cycles they can withstand before the solder balls become brittle.
4. I notice an increasing use of conductive glue to attach flat flex cables in computer parts, especially ultrabooks, and this also has a limited number of thermal cycles before it starts to degrade. (Anyone who bought a Samsung TV between 2005 and 2008 might already be aware of this.) This is probably the worst issue since it isn't going to be fixable for a hobbyist.
Also, I wouldn't even guarantee that firmware in Flash that isn't rewritten often will last 10 years. There are a lot of dead Nintendo Wiis already due to bad blocks in the flash.
I currently own a 30 year old PC-XT and a 40 year old stereo receiver, but they didn't make it to that age without maintenance and replaced components. I'm afraid that in 30 years many of today's devices will have failed in ways that are impossible to get at or repair.
Been there, record thunderstorm centered directly above a client's building. Several switch and mobo ports died. The fact that said client "saved" on cabling by running UTP between buildings probably had something to do with it.
Anyone who has ever wired up a T1 in the basement of a highrise knows what I mean.
-Matt
I buy for reliability. If I can plug something in and just forget about it (except for patches) for 3-4 years, we're done and whatever new thing I can buy will pay for itself in power savings.
I'm happy that an SSD will last that long, but it's not something I worry about.
I have had to rebuild servers because of abject SSD failure, and will no longer buy those brands. Failure seems to be quite highly correlated with trying to save a buck by getting non-top-tier drives (whereupon it's me, or someone like me, picking up the pieces when I could be writing code. Screw that).
If a drive was happily at 90% wear after two years, I'd just provision more of the same drive. Yay! :-)
-Matt
It's pretty much conjecture... I will say that I was pretty disappointed in a lot of the seagate 7200 3tb drives I bought 3 years ago.. out of 12, 3 are now dead under moderate use... (I bought in different batches from different vendors for teh same model).
Kingston has gotten very good (and they used to suck) at least in the eMMC space (not quite a SSD, but close).
I nailed one to the wall of our IT lab as a warning not to buy any more of them.
...at ninja swapping components under same model name, google V300
Not only is it going to give us some much needed IOP/s it's also going to save us hundreds of thousands of dollars on storage over the next 3 years.
If you're interested take a look at: http://smcleod.net/building-a-high-performance-ssd-san
firmware issues dominated SSD problems in the early years. Those issues are far less prevalent today though Samsung got its ass bitten by not adding enough redundancy and having data go bad on some of its newer offerings. Strictly a firmware issue. Which is another way of saying that one should always buy the not-so-bleeding edge technologies that have had the kinks worked out of them rather than the bleeding-edge-technologies that haven't.
If it starts to bite me I may change my tune. But until that happens I put it in the wasted-cycled category.
-Matt