I wonder whether (spinning disk) hard drives will still have a market in 10 years' time? I'd always assumed they were a bit of a dying breed and when the per-GB cost of SSDs reaches parity they'd become a museum relic...
What you'll see is that spinning rust will increasingly become used for cold(er) storage due to still offering cheaper bytes, and (relatively) ever increasing cost of IO (you get one spindle per il unit, but at increasing capacity, with barely a change in sequential throughout and no improvement in random access). This is happening in data centers now, either through explicit product tiring or clever offloading.
As the storage medium in this new tech is graphene, do we call it "spinning coal"?
It protects the storage medium (iron-platinum alloy), which cannot be used without a protective layer, due to corrosion and scratching risk.
SSDs are currently targeting performance because of the cost, but it seems possible to make them even more (and much more) power efficient when used as high latency storage.
For enterprise servers (I'm aware they are not relevant for AWS/GOOG etc), more and more often high-end machines are offered as SSD-only - for such price bracket SSDs price is already not an issue.
Unless you are me, who has had tough lessons at The School of Hard Knocks, with postgraduate degrees from The College of Getting the Shit Kicked out of You.
Not to mention that, with Cloud this and that, local hardware is pretty fungible and easily "reincarnated" in a replacement device
Re Apple: this is true only for desktops or laptops. If you want to back up your iPhone you need a Mac to work with a local storage solution. And you cannot configure the Mac to use the HD to store the phone backups directly: the folder the Mac uses cannot be a symlink nor can it be configured. You have to sync all of your phone's data to the Mac, then sync the mac with time machine, which means your Mac needs to be large enough to store all of its data, plus the iPhone's.
Source: my hackintosh has small SSD but large HDD.
I had SSDs dying gradually and HDDs dying abruptly too. I think this is just the difference between the controller hardware failing or the firmware encountering some unrecoverable condition vs. failure of the storage medium.
That said, there still is the difference that HDDs retain data much longer, flash cells leak charge over time. So for long-term storage HDDs are still a little better, but "long" here means many years, so you're still in the domain where you're risking age-related failure, redundancy will be required either way.
If you don't write to it and it's kept at a constant and low temperature/humidity and it doesn't get jostled then the data will probably outlast you.
I keep a plastic lap pad under my laptop when carrying it around horizontally. I see people just letting the whole torque of the laptop bend the motherboard and it hurts me. Those laptops will one day not POST because the CPU, RAM, or GPU will have a pad lifted off the PCB.
An SSD randomly giving the ghost is one of my greatest fears when it comes to desktop computers, because there's no way to get any data off them when they die. With spinning rust you can usually send the drive to an expensive data recovery company who will likely transplant the platters and send you a copy of what's on disk, but if a flash chip is slightly damaged because of a short or whatever, you won't have such luck.
Yes, I know, I should make more backups, but every time I thought I'd gotten everything safely backed up, I found a file or folder that I forgot to include.
I’ve had an nvme reaching temps 70+ celsius, because the geniuses who designed the mobo put the socket right where cpu hot air flows..
I aged a couple of years when the machine bsoded and then told me windows files are missing. I thought everything is gone, but fortunately when the ssd colled down everything went back to normal.
Had to put an extra fan in the case, and disable couple of cpu cores though to make the system stable.
Most of these quirks are undefined behavior, and you won’t ever see the source code.
That's the point.
SSDs keep achieving lower US$/GB, but so do spinning hard drives.
This means that the point in time when parity is reached is continually postponed.
A bricked NAND chip, on the other hand, is an entirely different kind of problem.
Magnets only lose their magnetic powers very slowly. For example, samarium cobalt magnets might decrease their magnetic strength about 1 percent over a decade, which contradicts your claim.
Most magnets made from cheaper materials with lower Curie temperatures, e.g. Nd-Fe-B or ferrites, lose their magnetization at higher rates at room temperature.
Losing a few percent of the magnetic strength for the whole magnet would be equivalent with massive data corruption on a HDD, because that is the percentage of magnetic domains whose magnetization has rotated, i.e. the percentage of corrupt bits. The magnetic materials used in HDDs have actually a much lower rate of spontaneous change of magnetization.
Nevertheless, when a single HDD contains around 10^14 bits, even a 10^-14 per year probability of a bit flipping at ambient temperature may cause one corrupt file per year of storage per HDD, which is close to the typical rates of corruption actually seen in reality.
Hard drives add ECC info to each sector to prevent random bitflips from corrupting data[1]. Also IME actual rates of data corruption is far lower than 1 bit per year. This is on a filesystem with checksums, so any corruption would be easily detected.
[1] https://en.wikipedia.org/wiki/File:Advanced_format_(4Kib)_HD...
I've only observed 1 incident of corruption in 5 years of 00+ drives with ZFS, and that was due to a dying HBA controller, not the HDD.
If the files are overwritten frequently, then the error rate would be much lower. Even if they are only read frequently that would lower dramatically the error rate, because the HDD controller will rewrite the blocks with corrected errors, which happen frequently.
In about 200 TB of high-quality HDDs from 2 vendors and in various capacities, between 0.5 TB and 8 TB per HDD, I have seen around 100 errors after 5 years of cold storage.
For materials with high coercitivity and at temperatures much lower than their Curie temperature, the probability is low but it is not null.
A few years ago, I went back to archiving my data on magnetic tapes (LTO). However, before that and after optical discs became to small to be usable as an archival medium, I have used HDDs for backup and archival, for more than a decade.
After returning to tapes, I have gradually transferred the old HDDs to tapes. I had a case with one HDD that would no longer spin after a few years of offline storage, but most functioned, at least enough to be able to transfer them on tapes.
All the archival HDDs were duplicated, for redundance, and all the files were stored with checksums to detect data corruption.
The checksums have been absolutely essential, because after several years of storage there were a lot of corrupted files, but for none of them the HDD reported any error. The HDDs were about half WD and half Seagate, all of them Raid Edition or equivalent.
I had a HDD archive with about 200 TB of data, and on average, after 5 years of storage, there was a corrupted file about every couple of TB.
However, because each HDD was duplicated and the errors did not affect the same file in both copies, I have recovered all files.
All the corrupted files were almost certainly caused by spontaneous magnetization rotation affecting a few bits, so what the previous poster said is not only theoretically correct, but proved in practice by at least one example.
Most people do not realize how frequent data corruption is on HDDs, because they do not use file checksums or extra file copies, to detect corrupted data.
Bit errors are frequent on HDDs, but most are corrected by the HDD firmware. However, from time to time the correction capabilities of the codes used in HDDs are exceeded and silent data corruption happens.
After a lot of experience, I never trust the firmware of any storage media, so I store checksums, in extended attributes, for all the files that I will keep for more than a day.
However, this behavior can obtained only by replacing the file system drivers with a custom version. That could be done on Linux but it cannot be done as a cross-platform solution.
So for cross-platform, you need a pair of scripts, one for verifying the checksums, recursively for all the files in a directory, and another script for updating all the checksums for the files in some directory, which you need to either invoke manually after editing some files or downloading some new files, or you could set the script to run periodically, e.g. once per day, maybe prior to a backup script.
Nowadays a bash script can run on any operating system, so it is cross-platform enough, but writing a new pair of scripts for each operating system for whatever tool is used there for scripting is not much work.
The 2 scripts should consist each of a pair of scripts, a top level script that gets all the files from a directory (e.g. on UNIX using 'find "$1" -type f') and invokes the second script for updating or verifying the checksums for a single file.
The scripts that do the work for one file just need to invoke some standard checksumming programs, e.g. sha256sum and so on, which are now available on any platform, if you take care to install them. For verifying, missing checksums or checksums that differ from a newly computed checksum give an error, for updating, the checksums are recomputed if they are missing or if their timestamp differs from the file modification timestamp.
In the beginning, I was storing the checksums in one extra file per directory, but that had many disadvantages. Then I have switched to the use of extended attributes, which allow the storage of the checksums together with the file, transparently.
Extended attributes are much more convenient, but their cross-platform use is more complex.
For updating or verifying the extended attributes, you need 3 commands, list, read and write extended attributes. These not only are different on each operating system, but they are some times different on a single operating system for different file systems.
So you either need to write different scripts for each platform, or you must detect the platform and choose the appropriate extended attribute commands. The work is about the same for both choices, so it might be simpler to just have different scripts for each platform. If you use per platform scripts, they can be small, less than a page each, so it is less likely to make mistakes in them.
When you use extended attributes, you must take care that on open-source systems, like Linux or FreeBSD, not all file systems support extended attributes and those that support them might have been configured to ignore them.
So, especially on Linux, you must check that you have enabled support for extended attributes and you might have to recompile the common utilities, like fileutils, because by default you might have commands like cp and mv that lose the extended attributes. Also neither all GUI file managers nor all archiver versions (e.g. of tar or pax), handle correctly the extended attributes.
The main pitfall on Linux is that tmpfs has only partial support for extended attributes, so on many Linux systems, copying a file to /tmp will lose its extended attributes.
When copying files through the network, the most reliable way is to use rsync, preferably rsync over ssh. That will copy any file together with its extended attributes, regardless what operating systems are at the 2 ends.
Otherwise, Samba can be used between most operating systems. I have used it in mixed configurations with Linux, FreeBSD and Windows computers, without problems with preserving the extended attributes.
There are new NFS versions with extended attribute support, but all the computers must be configured correctly, with non-default options. I have not used such a version of NFS.
So the conclusion is that the files themselves are cross-platform if you take care to not use for transferring them methods that would silently lose the extended attributes, but the simplest way is to rewrite a pair of updating/verification scripts for each platform, because those can be very simple, while a cross-platform script would be large and cumbersome.
You typed enough interesting stuff here for an above average blog/whatever post. Consider doing and submitting it.
ZFS gives me set-and-forget sha256 checksums that are kept up to date in as data is written in real time. And the checksums are block-indexed, so 100GB files do not need to be-hashed for every tiny change.
I also never have to worry about the "oh no which copy is correct" logistics nightmare intrinsic to A->B backup jobs in nightmare scenarios where they fail halfway through, as both copies of my data (I'm using 2-way mirroring at the moment because that's what my hardware setup permits) are independently hashed and ZFS will always verify both sets on read. (And reads are fast, too, going at 200MB/s on a reasonably old i3. No idea how any of this works... :) )
As for cross-platform compatibility, my understanding is that if the pools are carefully configured correctly then zfs send/recv actually works between Linux and FreeBSD. But this kind of capability definitely exceeds my own storage requirements, and I haven't explored it. (I understand it can form the basis of incremental backups, too, but that of course carries the risk of needing all N prior tapes to the current one to be intact to restore...)
The two mostly-theoretical-but-still-concerning niggles that I think a lot of setups sweep under the carpet are ECC memory distribution, and network FS resiliency.
All the ranting and raving (and it does read like that) out there about ECC means nada if the RAM in the office file server in the corner implements defense in depth (if you will), while files are mindlessly edited on devices that don't have ECC, through consumer/SOHO grade network gear (routers, switches, client NICs, ...) that mostly also don't use ECC. Bit of a hole in the logic there.
I did a shade-beyond-superficial look into how NFS works, and got the impression the underlying protocol mostly trusts the network to be correct and not flip bits or what have you, and that the only real workaround is to set up Kerberos (...yay...) so you can then enable encryption, which IIUC (need to double check) if set up correctly has AEAD properties (aka some form of hashing that cryptographically proves data wasn't modified/corrupted in flight).
SMB gets better points here in that it can IIRC enable encryption (with AEAD properties) much more trivially, but one major (for me) downside of SMB is that it doesn't integrate as tightly with the client FS page cache (and mmap()) as NFS does, so binaries have to be fully streamed to the client machine (along with all dependent shared objects) before they can launch - every time. On NFS, the first run is faster, and subsequent launches are instantaneous.
There’s also more exotic data storage like optical holography. That’s also 3D data storage, but with very little heat accumulation, which is the problem with anything based on 3D electronics. It is technically a spinning disk though.
And people have been saying this for a long time. I remember a few vocal members in certain popular Internet forum keeps bagging on about it in 17/18. ( And we are half way 2021 already ) There are currently no roadmap from both HDD and SSDs they will reach per-GB cost parity. Not in the next 5, and nothing that shows this will happen in next 10 years.
As a matter of fact there are plenty in the pipeline and R&D that could drive cost /GB on HDD cheaper. I dont see anything substantial with NAND. Other than the usual multiple layers and die shrinking. Both of them have fundamental issues with cost.
Imagine a world where a serious enthusast could fab say 32nm at home.
If it were to happen I don't think it'd look like an at-home-version of the present industrial process.
On that note, you could technically just buy the chips and spend an inordinate amount of time and money building the PCB and firmware.
Sure, if you want to make a 1 um process you can do that with a purchased wafer, some chemicals, and a spin plate. If you want to make a nm-scale process you'll need to have a serious budget.
And RAM is the new disk.