Absolutely mindblowing.
Absolutely mindblowing.
I'd like to expand. What I find mindblowing about it is that, as a regular consumer:
* When you need more space you can't just plug in another disk or USB stick. You also have to choose on which device you want to use it, and you have to tell all your software to use it. And that may involve shuffling data around.
* As a corollary, you need to remember in which device you put which stuff.
* As an extra corollary, any data loss is catastrophic by default.
* File copy operations still fail, and when they fail, they do so without ACID-strong commit/fallback semantics.
* Backups don't happen by default, and are not transparent to the end user.
* Data corruption can be silent.
Bonus, but related:
* You can't share arbitrary files with people without going through a 3rd party.
Similarly, many users don't have much valuable, unique data on their computers. The most important stuff very often lives completely in the cloud, where it's properly backed up, etc.
Also, the most ubiquitous computing device now is a smartphone. It has all the automatic backup stuff built in, you can put an SD card into it, and it will transparently extend the free space, without hassle. Even on PCs, MS and Apple nudge the users very prominently to use OneDrive and Apple Cloud for backing up their desktops / laptops. But past certain size, it costs money though, hence many people opt out of that. Again, because most people value the lowest price, and just hope for the best; because what could ever happen?
Silent data corruption can still be an issue, but, frankly, malware is a much bigger threat for a typical non-technical user.
Technical people have no trouble setting up all that: mount your disks under LVM, run ZFS on top of it, set up multiple backups, set up their own "magic wormhole" to share files with stranger easily. But they know why they are doing that.
Educating users about IT hygiene is key for improving it, much like educating people about the dangers of not washing hands, or of eating unhealthy stuff, helped improve their health.
Scholars will have more clue about life in 4th century via the Oxyrhynchus Papyri collection than 21st century Terra Haute, Indiana.
Most people tend to love going through their old photos, thinking of the people and places depicted, and sorting them. For an extremely long time this was entirely the norm and it took place mostly using large cumbersome physical books where meta data was recorded by hand by writing on the back of the photo.
It's become much easier now that files are digital, only slightly offset by the fact that digital has enabled us to take many more photos than in the past. What I see more often are people who have no idea when the photos on their phones are actually stored at all (what's a file system?), let alone how to copy them off their phone and onto something more appropriate besides resorting to something like emailing photos to themselves which has all the problems of adding and successfully transferring large file attachments.
It's a chore because companies really don't want people to have access to their own files unless it's on the company's terms and using their cloud servers.
If you're in the Apple ecosystem, please set up a recovery key so that your loved ones can access your photos and other things in case something unexpected should happen:
* https://support.apple.com/en-ca/109345
More generally: open Notepad (or whatever) and put the URL, username, and password for e-mail† account and bank/credit card accounts (and phone PIN), print it out, and put the paper in a sealed envelope and show its location (on top of fridge?) to a trusted friend/family.
† If someone can access your e-mail / SMS, they can probably reset any other account(s).
I know first hand the mess from my friend who died of cancer at 38 leaving behind a wife and daughter with not only only accounts being locked out but a sprawling home lab that they did not understand. Even things like smart lights became a stumbling block when they could not control them. The router was a virtual machine on a rack mounted server in the basement. Had to unwind a lot.
A possibly unpopular opinion is that people should think long and hard about things like smart lights so that they can have cool mood lighting or whatever. I'm needing to have a lot of electrical redone because of a kitchen fire--microwave offed itself in the middle of the night, scary--and I told my electrician "no smart stuff." Which he was totally on-board with. I honestly think it's pretty much a gimmick for most people.
It may be necessary for other people to get 'authorized' to get access to it.
If you have access to one, it's certainly useful in case something happens to you residence, but one should also probably have access to something more convenient (especially in the case of incapacitance).
There’s a contradiction here which I think is worth fully considering. People definitely do value their data – data recovery services have been getting revenue for decades from people who lost things they care about – but there’s been a market failure where operating system vendors left the users to fend for themselves, everyone agreed that was hard, and the majority of users quite rationally decided that cloud services were better than learning to be unpaid sysadmins.
What we should be asking is why this is all so hard. Local file systems with strong integrity checks and seamless replication onto attached storage should be so basic that people can just assume it’ll work without them having to spend time on it.
What you're asking for is basically a bug-free btrfs plus a bit of automation on top (unless you also want a smart caching layer, which we already know is a Hard Problem). That automation is simple enough that it would definitely exist by now if there was a bug-free btrfs for it to manage. But that turns out to be a hard problem, so anyone who wants to run their system with automatic add and remove and rebalance is looking for trouble. (ZFS has a better reputation on the reliability front, but is too inflexible for a consumer-grade pooling of arbitrary storage devices.)
Apple claims that their hardware makes it unnecessary for APFS but I’m skeptical and the last time I looked the Time Machine situation was still pretty clunky. I haven’t checked Windows recently since I only use Linux and macOS.
Apple was this close around Mac OS X 10.5. Time Machine is more-or-less a perfect backup solution for most people, and if they had ended up using ZFS rather than just testing it in a Server release, your average Mac user would have automatic block-level file hashing and verification (with background scanning and auto-recovery from backups with Time Machine, presumably) which is good enough for the vast majority of people.
APFS is an improvement on HFS+ but ZFS would’ve been much better. FileVault (2) may have complicated matters but who knows, maybe we would’ve seen sped up native encryption in ZFS with support from Apple.
Without ZFS (or some other automatic checksumming), Time Machine (or any other automatic backup solution of fixed size) backups with a good copy of data will eventually be updated with new corrupt copies of the data, and then aged out when the backup target runs out of space. The solution doesn’t have to be ZFS; my point was that it could’ve been, and very nearly was.
This is what makes automatic backups so important: people keep taking chances until something bad happens, and then they re very relieved when they find out that their work has been backed up somewhere. The same thing applies to cloud apps: they relieve people from the burden of caring about their data's safety and integrity. The iCloud backup that allows you to restore your lost phone down to the state of draft "unsaved" texts is the gold standard here.
I don't think this is an accident. These companies want you to hand all your data over to them so they can mine it for data, increase your reliance on them, and/or on paying them reoccurring fees to backup your precious files. They have a real financial inventive to make protecting your data without handing it over them very difficult, just like hard drive manufacturers have an incentive to sell you garbage drives you'll need to frequently replace.
Complete BS. Cryptolockers and similar are not a concern for regular users while disk failure is something pretty much everyone gets to experience over their lifetime.
To be fair some more, JBOD/RAID and hard drives does work pretty well. Past the 5 year horizon to be sure.
Product mgt and corp finance has also fallen in love with subscriptions - and clown storage is such an awesome match for that! Who needs to sell long term terabyte solutions when you can rent it out. Easy to argue against that logic of course, but not easy to fight.
I don't mind living in the command line and I don't even mind fighting to get everything up, but I don't see most people putting up with it. It's also a huge pain to get working with a laptop, since I don't think most laptops have a SAS connector, so you have to use an eGPU case with a Host Bus Adapter, which is its own share of headaches.
But that goes back to the minimal market for LTO among amateurs. They will scratch their itch and write software for it but it's not exactly critical mass.
I've gotten it to work over thunderbolt with an eGPU case on both the mini computer and my laptop, so it can be done, at least with NixOS.
I still have the tapes, and I use them to archive really important stuff, but honestly what I mostly do now is have a pretty resilient ZFS RAID, which is 24 16TB drives, three RAID-Z2's chained together, so I can lose up to six drives (two from each RAID) before I risk losing data. I run a scrub once a month, so if a drive dies, hopefully I can catch it before I lose anything.
If your datasets are in the 100GB range, Blu-ray archival might be a better option. The disks are supposed to last 100 years if well stored and the drivers are much, much, much cheaper.
That‘s getting near S-ATA III (SATA-600) limits.
Also a LTO-9 tape can store up to 45TB of uncompressed data (18 TB uncompressed), so you better have a fast storage volume ready with your data when you’re writing your next tape.
Long-term archival, however, is a different thing and tape is a much better medium for that.
It's a shame DOTS [1] doesn't seem to be going anywhere...
Clown storage.
Thanks for the giggle. :D Needed it, had a pretty rough last couple of days.
And NAND flash is already pretty 3D, with hundreds of layers of memory cells fabricated on the surface of each wafer, and several such dies stacked in each BGA package, and it's not uncommon for U.2 drives to contain two PCBs each with NAND packages on each side.
Another way to look at it: "and yet here we are". Tape is still what has led to the highest density so far (depending how much storage you look at, and cost tradeoff). But also cost tradeoff of separating the drive from the medium. which hasn't really worked: LTO tapes are pro supplies - and so, expensive.
Solid state electronics and magnetic media are beyond magical. The odds of keeping terabytes of data on rails are astronomically bad.
I have a prototype first chapter written that cold opens with an archeological dig '...John Li Wei looked up from his field journal where he had just written “No artifacts found in Basement Level 1, Site 46-012-0023”, wiping sweat from his brow. "Did you find something, Arnold?" he asked, his voice weary. "Three days in this godforsaken jungle, and we've got nothing but mud to show for it. Every book in this library’s long since turned to muck.” Arnold gestured towards the section of the site he had been laboring for the last 30 minutes, digging through layer after layer of brown muck, with fragments of metal hardware that once supported shelving. A glint of metal caught the filtered light. “Arnold, that’s just another computer case,” John sighed, his shoulders slumping slightly. He could already imagine the corroded metal and the disintegrated components inside. Useless. “Help me pull this out.” The two men strained against the clinging earth, their boots sinking into the mud with each heave. As they finally wrestled the heavy, corroded metal case free, a piercing shriek cut through the jungle sounds – beep, beep, beep, beep....'
This is another portion of later in the first chapter.
'...The transport, a ground-based vehicle, levitated silently outside. John glanced at his chronometer as he boarded. Jeg er ked af det, he thought, I'm late. The doors hissed shut, and the vehicle computer announced, “Destination: University. Estimated arrival: 25 minutes.” With a gentle hum, the vehicle glided smoothly along the elevated guideway. The air inside was cool and faintly metallic. Outside, the landscape was a patchwork of green fields, managed forests, and gleaming white research facilities. The transport’s progress was slow; the gentle sway and hum of the engines were a constant reminder of the NAU’s strict energy policies. John sighed, thinking about the upcoming lecture. How could he convey the importance of the Bit Rot Era when so many were focused on the pressing needs of the present?
At the University, John rushed into the classroom, three students already waiting. “Jeg er ked af det, I’m late,” John said, a slight Danish accent coloring his Danglish. The classroom was intimate, a very different design from the great lecture halls of antiquity, designed for perhaps twenty students, with a central platform surrounded by holographic cameras for remote attendees. Historical maps and timelines adorned the interactive displays lining the walls.
John quickly moved to the lectern and carefully removed six artifacts from his bag. “Velkommen to Ancient North America 1,” he began. “Welcome to Ancient North America 1. This is the class where you are going to learn about the past that was and the present that may yet be.” He gestured to a holographic timeline that appeared above the lectern. “In broad strokes, we consider the history of Ancient North America to consist of four periods: the pre-colonial, the rise of the nations – of Canada, the United States of America, and Mexico – the pre-Collapse, and the post-Collapse periods. You can take in-depth courses on most of these. Dr. Jones’ ‘Rise of the Nations’ is well worth your study to learn about the United States, its Constitution, and Canada. This very university is located in a place that was once called Newfoundland, Canada, and in ancient times the climate was quite harsh. Meget forskelligt from the lush green farmland, forests, and even nice beaches we have today. You can also take Dr. Pech’s history of the pre-colonial tribes and empires. That will teach you a great deal about the people who first inhabited the continent and learn about their history, culture, foodstuffs, many of which are still considered extinct. Desværre.
“However,” John continued, his tone shifting, “the class you cannot take is for what we call the Bit Rot Era. A ‘bit’… imagine a light switch. On or off. That’s a bit. One or zero. The basic unit of digital information. The Bit Rot Era, from roughly the 20th to the 25th century, is a complete black box. En sort boks. What we do know comes from fragments of writing on paper. Millions of books were printed, but even those are often lost to time. We have fragments of ancient texts that talk about computers in every home and the ‘digitization’ of information and libraries. Digitization was the process of scanning physical media into computers. We’ve recovered millions of artifacts – disintegrated polycarbonates, silica, and bits of rare metals that were once these computers. But nothing within survived. Then something happened. The books vanished. The period from about the 21st to the 25th centuries was, as it were, expunged from history. Research indicates they went entirely digital, depending on system administrators to maintain the data… until the administrators stopped.”...'
Most web content I've consumed in my lifetime is already lost, many floppy disks and burned CD-ROMs I held onto are now unreadable, and in 200 years, the situation will only be worse.
But I can go to the British Library and read a 1000+ year old text without much difficulty.
From: https://archive.org/about
> A single copy of the Internet Archive library collection occupies 145+ Petabytes of server space -- and we store at least 2 copies of everything).
Thankfully they have a few backups around the world, but there are still plenty of plausible events which could render the data useless. In contrast, physical mediums like books, vinyl records or microfilm/microfiche can last for hundreds of years.
A lot of old printed books were done on acidic paper, causing them to more easily disintegrate. Disintegration of acidic books can mitigated by brushing alkaline powder between each page, and by keeping very low humidity.
The switch to alkaline paper substantially changes the smell, much to some readers' dismay. But be sure to look for alkaline prints, when available.
For example, using something like ggwave [1]. I guess that would last way more than 5 years (although the data density is rather poor).
A LTO8 tape stores 12,000,000 megabytes.
"Rather poor" is putting it mildly. This sent me down a sort rabbit hole. From a Stack Exchange discussion[0] it was a short trip to exceedingly technical discussion about using QAM encoding[1] to really beef-up the storage capability.
With the wacky QAM encoding tt looks like maybe 20MB per C90 cassette (and 90 minutes to "read" it back).
[0] https://retrocomputing.stackexchange.com/questions/9260/how-...
I would be careful with that, however. If you are archiving your data, it's because you like it and, if you like your data, you want it to be readable a long time from now. I'd suggest vinyl records rather than tapes, as they are very robust, and can be read without physical contact.
Part of that may be the tape drive (about 40 years old) but the reality is that consumer level cassette tapes aren't built to last: magnetic fields weaken, coating flakes off, tape stretches, and other factors prevent these from being storage solutions beyond 10-20 years (my guess), if that.
They might be a fun nostalgic diversion for listening to old music, where the audio degradation is part of the experience, but for data, they're a non-starter in my book.
So it's up to us really. I have a fairly OK setup, one copy on a local machine and several encrypted compressed copies in the cloud. It's not bulletproof but has saved my neck twice now, so can't complain. It's also manual...
We the techies in general are dragging our feet on this though. We should have commoditized this stuff a decade ago because it's blindingly obvious the corps don't want to do it (not for free and with the quality we can do it anyway). Should have done app installers for all 3 major OS-es, zero-interaction-required unattended auto-updates -- make is to grandma would never know it's there and it's working. The only thing it asks is access to your cloud storage accounts and it decides automatically what goes where and how (kind of like disk RAID setups I suppose).
While I do appreciate the generosity in how many projects make themselves available (free or otherwise), it does seem like they can have a narrow focus where they solve the challenge they had to solve, but aren't interested going past that point. There's logical reasons for why that happens, but there's unfulfilled potential to make personal computing a better environment there.
Oh absolutely, I agree so much with this. I get it, we can get obsessed with UNIX philosophy at times, one tool does its job perfectly etc. but we don't even make an attempt to assemble several things into one good cohesive whole.
Sadly I am way too swamped with being out of a job and chronically ill but I just started becoming extremely bitter towards many more privileged techies who seem much more interested in farting out the 2489th LISP interpreter than solve a real world problem. :|
- Management of backups themselves: borg / restic (or rustic) / duplicati / duplicacy etc. Last I evaluated them borg offered the most in one package. I like restic a bit more but it's a fair bit slower than borg, sadly. `rustic` aims to fix that but it's a volunteer effort and the author has bursts of productivity that are not on a stable schedule. I use rustic as a backup of my main backup tool (borg) but for now dare not use it exclusively.
- Management of keys / secrets / passwords: my semi-vulnerable setup is to just upload my borg repo key to my Linux server and also have it pinned in one of my private Telegram channels. That one could be done better but I haven't done research. Probably something close to gpg (many modern alternatives with less friction but I forget names; I have them bookmarked though) to make sure even your keys are properly protected. One possibility is a password vault like keepass[x] or Enpass or others.
- Management of storage: I use a local Linux server (EDIT: I have a very non-redundant and basic ZFS setup) and 5+ cloud storage services on the free tier. All of them have at least 5GB and my borg repository is barely 150MB I think. That includes ~20 past backups.
The way these could be combined is mostly by having a top-notch GUI and CLI (so we serve different kinds of people) that allows for granting access to cloud storage servers, asks optionally for a local storage server (NFS, Samba, WebDAV etc.) and just does everything else by itself. It's very doable.
Regarding the management of storage, what do you mean? Kopia already supports different "storage backends" out of the box which can be configured via the GUI. Do you mean that you would like it to be able to merge the storage of different storage providers, so that you can use multiple free tiers to get the storage space you need?
So you will need a 40 year old drive by then... good luck finding one that works.