Fujifilm Created a Magnetic Tape That Can Store 580 Terabytes
petapixel.com
petapixel.com
> Data on the tape is stored at a record-breaking density of 317 gigabytes per square inch...
> When tape is being read it is streamed over the head at a speed of about 15 km/h and with our new servo technologies we are still able to position the tape head with an accuracy that is about 1.5 times the width of a DNA molecule.
Me wondering about those micro SD cards that can store 1Tb.
Ad absurdum, stacking two Micro SD cards on top of one another hasn't just doubled your density even though surface area is unchanged.
In theory, write-once memory cards, using some kind of antifuses, could be designed to have a lifetime good enough for archival storage, but nobody has attempted to develop such a technology, because it is not clear if there would be a market for them.
Most people do not think far ahead in the future, so they do not care much about archival data storage, until it is too late and the information had already been lost.
I disagree that it's unusable. You'd end up with a puck the size of a data tape that can archive a petabyte of data and needs to be plugged in to a 5 watt power supply for long term storage. That's not super onerous. Then consider that tapes need to be stored at exactly room temperature with 20-50 percent humidity, while this puck would barely care about environment at all. And you could plug it directly into a computer without a $5k drive. Honestly it sounds pretty good to me. We just need to drop the price of flash by a factor of 20 to make the scenario happen.
“Just let me geek out for a second”
Absolutely IBM storage person, the more geeking out the better!
The tape stores 580TB for a length of 1255m, does that mean the read speed is 580TB/1255m*15km/h = 1.9TB/s ? Seems too high.
It did say that. I wonder if there's some multi-track/double-layer/double-sided stuff going on that needs multiple passes to read, that does sound awful fast!
Also, I suppose they limit the read bandwidth with something like what a single Infiniband connection would support; few disk arrays support much higher speeds.
You could hypothetically do that, for sure, but the software basically is derived from the tape era where you'd just have one logical stream coming out of the tape.
The drive has 32 heads, and reads/writes 32 tracks at a time. It goes from the start of the tape to the end, then aligns with the next 32 tracks and reverses direction.
Each group of 32 tracks is called a wrap. There are 208 wraps, so 6656 total data tracks. Even wraps go one direction, and odd wraps go the other direction.
That's the important part.
But also the tape is divided into 4 "bands", each one holding a quarter of the wraps/tracks. Between the bands, and at the edges of the tape, are special servo tracks that are used for alignment.
So when a source talks about "wraps per band", it's pointless abstraction. Unless you're really in the weeds, the only thing you want to know is the total number of wraps.
The numbers they cite for "previous generations" don't match up to the progression of the LTO tape series' capacity. Maybe they are citing "research numbers" that they can do in a lab but aren't production-ready yet. I would certainly assume they are citing "compressed" data figured there.
Also bear in mind that tapes typically store data striped across the tape in multiple tracks and multiple bands. There are four bands per tape and 12-52 wraps per band, so reading the whole tape requires up to 208 passes across the tape.
https://en.wikipedia.org/wiki/Linear_Tape-Open#Physical_stru...
But yes, to agree with another parallel comment, tape data rates are quite high sequentially (abysmal in random of course, but that's not how they're used). LTO-8 does 750 megabytes per second compressed / 360 megabytes per second raw.
Crazy good tech accomplishment right there.
In a way, I am somehow comforted. Tape remaining relevant kind of dropped off my radar. Good to see.
317Gb / square inch AND 15km/h transport speed? That is nuts fast.
Wonder how robust it really is? Of course at those densities and potential transfer rates there is ample room for error recovery.
Not quite sure where IBM's numbers here come from, their previous numbers don't match up to the progression of the LTO tape series' capacity. Maybe they are citing "research numbers" that they can do in a lab but aren't production-ready yet. I would certainly assume they are citing "compressed" data figured there.
But certainly tape has continued to progress much faster than most people would have imagined. Big tape libraries are still a thing in certain environments and they work very well, there is no better solution for bulk cold storage.
LTO is cool and all but is the "compressed capacity" number really something to repeat with a straight face? The tape holds 18TB, we don't need to pretend it's anything else.
> But certainly tape has continued to progress much faster than most people would have imagined.
Mostly it has. But I'm somewhat worried about the future after the sudden late-game announcement that LTO-9 would have a 50% capacity improvement instead of the usual doubling.
No, but it's been the standard in the tape industry for decades. Probably dates back to the first tape controllers that had builtin compression (so compression didn't tax the main cpu).
IRIX allowed for live file system backups and the drive doing compression meant all that happening with negligible user performance impact.
Was literally set it and forget it, aside from tape rotation into off site storage.
Compression would have had an impact.
We don't do multi user app serving much today, so maybe a smart drive has less benefit. But it mattered then. 2000's era.
The hardware is not going to read 300 Gb/in density at bicycle speeds. :)
That seem perfectly reasonable. You can get 256 USB flash in in size about half square inch, and that is random access data. (e.g: https://www.walmart.com/ip/SAMSUNG-256GB-Bar-Plus-Titan-Gray... )
The table from IBM[1] in the middle of the page says this number is in gigabits (not gigabytes) per square inch. This is obviously impressive nonetheless, but I wonder what else this article got wrong.
[1] https://petapixel.com/assets/uploads/2020/12/ibmtale.jpg
Yawn on the raw density figure, though.
The chip inside a 128 gigabyte micro SD card is a small fraction of a square inch and is cunning enough to provide random access.
You just can't easily and cheaply have a long tape of them.
It used to be the case once upon a time that mass storage media like magnetic tapes and discs based on writing on surfaces with a head had better density than memory chips.
It was faster and more secure since we were working on air gapped networks.
If the courier in question is 6'2" that might be the case, but I suspect if the person was much smaller and is a heavy sleeper, that it wouldn't be too bad.
I’m 1.97m or so and it’s really not designed for me. I suspect it’s optimised for those at 1.6-1.7m.
Or, maybe, even a Zen monk who spends time meditating.
I agree with OP, if this was a first class flight, it wouldn't be a bad gig.
Of course you don’t check it if it’s that critical.
Network volitility is a huge problem when transferring huge files.
Checksums will identify something went wrong, and then you need to redownload the file to a quarantine network and scan it. Takes time.
Much easier to go from trusted source to trusted source and verify the files on drive prior to shipping.
Amazon just released a device (forget the name) for exactly the same use case. We developed it in house.
Not to mention most major studios will contractually prevent you from exposing anything to the web.
>Checksums will identify something went wrong, and then you need to redownload the file to a quarantine network and scan it. Takes time.
Surely a sensible file transfer algorithm would compute checksums on small and easy-to-retransmit chunks? Does rsync not do this? Isn't it already happening in TCP?
>Not to mention most major studios will contractually prevent you from exposing anything to the web.
I understand that workstations with media on them are not going to have internet access, but do they really prohibit site to site VPNs?
>Amazon just released a device (forget the name) for exactly the same use case. We developed it in house.
Snowball and Snowmobile? IIRC these are primarily meant for one-time migration from on-prem storage to S3. Do people really use them on an ongoing basis?
I'm envisioning a huge device in the back of a van which pulses a powerful beam, similar to typical movies (Oceans 11?) cutting the power to a bank / casino prior to a raid
It would probably take more power than that to do what is being suggested from the distances that are being suggested.
I'm not sure it's feasible.
That said, I was able to recover 90% of the voice recordings my father made between 1959-1963 on reel-to-reel tapes 60 years later. Tape can be very durable but what I recovered was analog voice very tolerant of errors. I'm not so sure about gigabytes snuck into an inch-square.
First off, the drives themselves are going to be expensive brand-new. You're going to be paying thousands of dollars for a drive, and it's probably going to have some weird interconnect that you'll have to spend even more money and waste a PCIe slot on an adapter in order to use the drive. Most common is SAS or Fibre Channel; although there's at least one company selling drives in Thunderbolt enclosures for the Mac market.
(Aside from all that, SAS is actually pretty cool for things like hard drive enclosures, since it has SATA compatibility. I have a jury-rigged disk shelf built out of an old HP SAS expander, a slightly-modified CD duplicator case, and some 5 1/4" hard drive bays.)
Second, tape formats are constantly moving. LTO and IBM 3592 come out with a new format every 2-3 years and backwards compatibility is limited. Generally speaking you can only write one generation behind on LTO and read two generations back. So, if you want a format that's got drives still being made for it, you'll need to migrate every 5-7 years. Sure, the actual tape is shelf-stable for longer, but you're going to be buying spare drives or jumping on eBay if you want to keep old tapes around that long.
(eBay is actually not a bad place to buy used tape drives, but the pricing varies wildly. It's perfect for hobbyists and small-fry IT outfits looking for cheap backup media. Absolutely terrible if you're a large outfit with reliability guarantees and support agreements to maintain.)
Third, actually using a tape drive is a nightmare. First off, Windows hasn't shipped tape software since 2003 (I think?), so you'll be in the market for proprietary backup solutions. Second, if you're just writing data directly to the tape, you will shoe-shine for days. Common IT practice is to actually have a second disk array sitting in front of the tapes as a write cache and custom software to copy data to the tapes at full speed once all the slow nonsequential IO is done. Reading from tape doesn't have to worry about this as much, but the fact that you had to use custom software just to archive your files means that you now have proprietary archive formats to deal with. So you can have tapes that rely on both having access to working drives and licenses for proprietary backup utilities.
(Of course, if you had decently fast SSDs and a parallel archival utility, you could sustain decent write speeds on tape. I actually wrote this myself as an experiment: https://github.com/kmeisthax/rapidtar and it can saturate the LTO-5 drive I tested this with.)
(I believe 8" floppy drives have the same interface as 5.25" ones --- and there's no shortage of adapters from the retrocomputing community for those, some of them even open-source.)
Tape is far more closed, AFAIK most of the common formats are proprietary and the specs are behind NDAs and other walls.
Every 10 to 15 years, you send in your archived (and new/interim) personal data and get it back on the current top-tech storage medium. That way it's not stored in the cloud and you can keep moving the stored data forward without having to deal with it all yourself.
Also, when others inherit the media they may not have proper equipment or skill to do it themselves as goal of the preservation is to get it 100 years ahead, not keep it always in a usable state per se. For example, I'd like my children to keep my backups until my grandchildren could access them 60 years later.
For book keeping, I think my point is that storage media are becoming so big that you always consolidate into a single device every time you carry the data over (you may still want to duplicate for reliability). Like you can buy a 18TB hard drive today. A consumer isn't going to require more than one or perhaps two of those for anything to be preserved long term. And in 5y-10y, you will likely have 25-30TB hard drives.
The equipment problem is precisely what this addresses. You are always using the latest hardware, and the previous hardware you are using is still supported if you stick to a 5-10y cycle. For instance you would have moved away from IDE drives while you could still find motherboards with both IDE and SATA ports. But if your data is stored on an IDE drive, good luck connecting it on a computer in 2030 (if we haven't moved full Apple's "you can't customize your hardware and we deprecate everything very frequently").
Skills (and I would say mostly dedication) is still a problem. But we are talking about copy-pasting files between two media, it's not rocket science even if you don't script it.
A quick google search brings up amateur-ish blog posts, and even those co sign the medium.
> The result of this study is that the GlassMasterDisc has a much longer lifetime in accelerated aging than other available DVD±R
I wouldn't draw any other conclusions on normal ageing of other tested media. They did an accelerated aging test at 90°C and 85% RH, where most discs didn't last a single test cycle (of 10 days), two discs lasted a single cycle, and only syylex lasted all 4 cycles.
Quote on a brand-name DVD
> This DVD model had the longest lifetime (i.e. 1500h) at 80°C and 85% RH. At 90°C, it is destroyed after the first cycle of 250 hours.
For an idea of what it does to the substrate:
> [for measurement] DVDs have to be taken [out] .. To prevent the formation of water droplets in the polycarbonate, it is necessary to "purify" the polycarbonate from the water that was absorbed at high temperature.
OTOH, I had CDs (Verbatim, upper middle class), of which about 1-2 of 50 had issues after 20 years storage (in dark, mostly room-temperature conditions).
It was a pupose-built test to see how much of X would Syylex take. It took X better than others, none of which took X well. Tests like these are very good, if you want to go with Syylex, to make sure it's not worse in some way (X, or Y, Z), which would then suggest a need for further examination. In real aging, factor X may be completely meaningless, while Y and Z are crucial, so you cannot conclude which one will last more.
Why test 90°C and 85% RH, not 80°C, 50°C or 110°C, or bending, UV light, scratching, drop in acid .. whatever? For a proper accelerated lifetime test, you would need to identify (all) relevant degradation modes and model their behaviour (and interaction) in target vs. accelarated test conditions, and then extrapolate behaviour in target conditions. They didn't even write what type of degradation they are testing.
I realize for the hacker news audience there are multitudes of reasons the solution above doesn't fit your needs, but realize the consumer market is near nonexistent.
If you wanted to make a sort of digital time capsule and didn't care who discovered it, your next best bet would probably be the Internet Archive or some other archival community.
If your data isn't appropriate for archival (i.e., can't be publicly consumed) and isn't interesting enough for your friends/family/etc. to keep around on your behalf, keeping the data is purposeless.
Though considering these comments and the advent of mRNA/CRISPR, perhaps we could store data for future generations in our own DNA. That'd be fascinating if you could read journals or even audio/video of your ancestors from your biological inheritance from them. What if we could engineer an extra chromosome to do just that, then let them remix and recombine segments of memories so everyone's would be unique.
By comparison, we'll create more data in the next ten minutes than entire centuries from our relatively recent history. Lots of stuff is getting preserved in lots of places with substantive redundancy for virtually nothing. Your slippers today are likely to be more valuable than the near-infinite troves of documents and photos and whatever else.
A single bad bug or security issue can make data inaccessible or corrupt. Tapes on the other hand does not have that issue. IMHO trusting all your important data to a single vendor or technology is a recipe for disaster.
Use an M-Disc with 3x PAR redundancy.
Or RAID 1 enterprise magnetic HD with 3x PAR.
Or all of the above :-)
Are you trying to archive TBs or MBs? Likely the next most important consideration is a stable place for storage.
Since I have a lot of stuff on M-DISC, you’ve now got me worried. Link? Any other info? I don’t need 1000 years but what is the lifespan?
250 hours @ 90 C°/85% humidity, how does that compare to - say - 100,000 hours at a "normal" 10-40 C°/30-50% humidity?
IF my last guess in comparison is correct, 100,000 hours at "normal" temperature/humidity is roughly 11 years, but it may well be that without "cooking" them at 90 C°, the duration is for both 200,000 hours (or whatever) ...
Single point anecdata, I had to dig i
To my surprise the pitch of the samples was a little lower (and varying ever so slightly over the duration of the song) than what you’d have expected with a440 tuning. It baffled me, since I expected some of the early digital synths used in the original sessions should habe been rock solid 440 tuning.
And that’s how I learned about “tape stretch” where analog audio tape stretches just enough to make the pitch of everything a few cents lower over long period of time.
p.s. I ended up applying digital pitch correction, so I could “jam along” with my own synths :-)
A better way would be to use something like that to store it for a decade or two, then copy the data onto whatever the newer version of archive medium is (LTO drives can typically read the last one or two versions as well). Rinse and repeat every decade, and it also lets you test if there has been any bitrot.
I currently have 8TB of data stored on a Synology DS218+ with RAID1, and monthly data scrubs (verifying checksums). It is backed up remotely to Google Drive (in encrypted form), and I also maintain an infrequently-updated, once-per-quarter disk clone with an external HDD.
My biggest concern with my current setup is that the memory is non-ECC. Even though the files are checksummed, I am concerned that specific memory corruption / bit-flips could propagnate into the checksums, and hence result in data corruption.
I am considering:
* Building my own FreeNAS box using AMD Ryzen (which semi-officially support ECC memory). My concerns here are the semi-official nature of support: how do I know if ECC works, before a rare cosmic bit-flip?
* Purchasing a Synology DS1621+. This is AUD$1400 which is a tough pill to swallow, for the equivalent of a first-gen Ryzen quad core and 4GB of memory.
Any options or recommendations is appreciated!!
The other thing I am interested in is minimizing idle power consumption. Just to be more environmentally friendly.
I use my Truenas box for storage using ZFS, VMs and NFS server for different PCs.
I bought ECC memory as I understand this is more or less a requirement for ZFS.
I found out that FreeBSD which Truenas is based on can give you info about what type of RAM is present.
The command is: # dmidecode -t memory
According to this I have ECC RAM. :)
I did a build with the following parts:
Case: SST-CS380 V2 (space for 8 3.5" drives, 2 x ). Mainboard: ASRock X470D4U2-2T Power supply: Seasonic Focus Plus 550W Gold 80 Plus Full Modular Power Supply RAM: Kingston Server Premier, DDR4, 16 GB, KSM26ED8/5 M CPU: AMD Ryzen 5 3600X Wraith Spire CPU Cooler: Arctic Liquid Freezer II NVMe to PCI bridge: ASUS Hyper M.2 x16 Gen 4 (PCIe 4.0/3.0) supports 4X NVMe M.2 devices (2242/2260/2280/22110) up to 256 Gbps for AMD TRX40 / X570 PCIe 4.0 NVMe RAID and Intel® RAID platform - CPU features
I bought all this from Amazon.de. The main board is a server main board with 10 GE Ethernet and a console for flashing bios and remoting into the machine - no graphics card needed. This was expensive and you can most likely save a lot using a consumer grade main board.
The RAM was taken from the list provided at https://www.asrockrack.com/general/productdetail.asp?Model=X...
The CPU models supported are here: https://www.asrockrack.com/general/productdetail.asp?Model=X...
Be careful not to buy an AMD APU - this doesn't support ECC RAM for some insane reason. An APU would have build in graphics and the CPU.
I use both SATA drives (long term storage) and SDD drives (for speed).
I created two ZFS volumes (I don't remember the proper terms). One for rotating discs (which could sleep most of the time) and one using SDDs for fast storage which doesn't use much power.
I have 6 KW solar cells with battery so I don't really care if the box uses a lot of power. During daylight its more or less free when sun is shining. I get next to nothing when selling the electricity that I generate and would like to use as much as possible locally.
Go with option 1, and Raid-Z2+. With RaidZ2, you’ll be able to not only detect but correct a flipped bit - Even if that flip happens when writing out the data.
Pay attention to the counters. Your ZFS scrubs will report how many resilvers they have. You’re likely to encounter 1 in a scrub. You’re unlikely to encounter more than 2. If you see that, that’s when you check for memory errors. A single bad sector is likely the hard drive. Even a single flipped bit is likely a transient error; It could be your memory, or your disk, or anything in between. It happens at scale, and 8TB, read repeatedly, is a lot of bits.
Look into rasdaemon and memtest86 - They’re the tools you use to debug what happens when
The other advice I can give you: Don’t be paranoid. Your photos are likely to acquire bit rot. You will have dozens, even hundreds of bit flips that will happen in your lifetime. Of the many thousands of photos you will take, the chances that you will ever notice the discoloration or bad line that a bit flip will have in a photo are pretty small. Bit rot happens. Your photos are important to you, and you should treasure them, but treasure them for what they are: Things that you protect and are under your care, not things that must be twenty nines of correct. You can realistically achieve 10, even 12 nines of correct reads on your data. You don’t need more.
"Next, you read a copy of the same block – this copy might be a redundant copy, or it might be reconstructed from parity, depending on your topology. The redundant copy is easy to visualize – you literally stored another copy of the block on another disk. Now, if your evil RAM leaves this block alone, ZFS will see that the second copy matches its checksum, and so it will overwrite the first block with the same data it had originally – no data was lost here, just a few wasted disk cycles. OK. But what if your evil RAM flips a bit in the second copy? Since it doesn’t match the checksum either, ZFS doesn’t overwrite anything. It logs an unrecoverable data error for that block, and leaves both copies untouched on disk. No data has been corrupted. A later scrub will attempt to read all copies of that block and validate them just as though the error had never happened, and if this time either copy passes, the error will be cleared and the block will be marked valid again (with any copies that don’t pass validation being overwritten from the one that did)."
(dont just read the quote, read the link)
I am using ZFS since 2007/2008 and I have never had any issues (except with those damn Seagate 3tb DeathStar hdds, where I was barely able to replace them fast enough - in 3 months 3 were gone - i will never buy seagate again)
Co-creator of ZFS: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
I am having a asus microatx board with 16gb of non ecc ram, additional SAS HBA and 3x3Tb Toshiba into zraid and additional 10TB HGST He3 disks + Ultrium 3000 (LTO-5, they are quite cheap today and are designed for 24/7 operation which is barely what I will ever encounter while tapes can be restored on latest LTOs if needed, while you can get cartridges on some sale for peanuts) for backups. There is no way in hell to go for my important data (like images) to disk only and tape is nice. You take the cartridge and store it at parents/gf/worspace drawer/...
Anyway if I remembered correctly, google has lost 150k of user accounts in 201x and restored them from tapes. So even for cloud opinionated people, it still makes sense to shovel important data to tapes if you dont use the data in everyday processing (and just an info - even shelved disks die)
Turns out it might not be that outdated after all.
It was my own home videos. I wanted to preserve them, so wanted to upload them to Google Photos.
It took me some looking around in eBay to find a camcorder to play these back. When it came home, I realized that I needed a FireWire 400 port to capture full resolution. So more digging around for a FireWire PCI-E card. I was finally able to transcode and upload some 15GB worth of video. It took 3 minutes on my gigabit internet for the upload. About a month for the whole process acquiring the hardware etc.
When I think about these tapes some 50 years from now, it might as well be completely unrecoverable, not because the tapes went bad, but because we have nothing to read them. Makes you wonder about galactic time scales like in Star Trek.
Here's where I raise my hand and ask "Why?"
They've been sitting in a box for 15 years. You never watched them, you never even thought about them. Why preserve them?
It's why I stopped taking pictures and videos of things. I never watched them again. It's all just a lot of waste motion over some dream that someday we'll find value in sitting and looking at this old stuff again.
I’m itching to do the same to my parents’ collection so my kids can see more.
I suspect their kids will enjoy the same.
At HN we assume best intention, asking why is how we learn about other people and their reasoning.
Of course it may be different than ours. But isn't that how we learn and are exposed to new ideas?
IMHO we should encourage more respectful "why?" questions.
> It's all just a lot of waste motion over some dream that someday we'll find value in sitting and looking at this old stuff again.
>It's why I stopped taking pictures and videos of things. I never watched them again. It's all just a lot of waste motion over some dream that someday we'll find value in sitting and looking at this old stuff again.
The author uses "I" over and over again, explaining that they don't see the point, so asking about a different perspective seems genuine to me, or at least there's a plausible interpretation that it is genuine.
From HN guidelines:
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
It’s like asking why someone would want to spread ashes or preserve an old piece of rickety furniture a long lost relative built.
Wait until you have to clean out a parent's house with a rooms full of stuff saved because they thought the grandkids might find it interesting. Here's the reality: they don't. My mother saved boxes and boxes and boxes of photos. None of them ever sorted, or put in albums. Just thrown in boxes. Saved for decades. Was it hard to throw them away? Yes, a little. There were a lot of moments of my childhood there. But if I asked myself honestly, was I going to do anything with them other than put them on a shelf at my house? The answer was no. I'd advise sparing your kids that burden.
A friend of my mother's had saved every canceled check she ever wrote. Boxes of them, because she thought her kids or grandkids might be interested in them some day. Their reaction was likely: What's a check?
If you really cherish something, and it regularly brings happiness to your life, by all means save it. But do it for yourself, not for what you think your descendants will find interesting. And if it's been in a closet for 5 years, ask yourself why you are keeping it.
The largest issue with tape is that the drives themselves cost absurd amounts, and you better not cheap out because failure is both time consuming and scary.
Swapping tapes continues to be human intensive and restore times long. But the tapes themselves are so cheap that at this scale it becomes worthwhile again.
I'd be surprised if you could even get tapes for it that aren't sketch grey market stuff that you wouldn't want to use for backups anyway.
The tapes are pretty cheap at $8 on Amazon if you buy 20.
Those tapes only hold 200 GB, and you can get a 10 TB hard drive today for less than $300:
https://www.newegg.com/red-wd101efax-10tb/p/N82E16822234407
While tapes are theoretically cool, the the drives are just too rare for them to be of any practical value to a home user. Even if the media has a better archival life than a hard disk (say 50 years), it won't do you a damn bit of good if there are no drives available in 50 years to read it.
IMHO, hard drives are better for backup (even offline backup, just get a hot-swap bay and imagine the drive is tape [1]). Archival is a harder problem, but I've settled on using high-grade optical media, burned slow for fewer errors, making a bunch of redundant copies. Even though the media might not last as long, I'm pretty much guaranteed to be able to find a drive for the next several decades [2].
[1] "pretend this hard drive is a tape" is even an enterprise product: https://en.wikipedia.org/wiki/RDX_Technology
[2] For instance, you can still attach a 5.25 floppy drive to a modern computer: https://www.ebay.com/sch/i.html?_nkw=5.25+floppy+drive + http://www.deviceside.com/fc5025.html
That was a link to an obsolete LTO-2 drive. If you’re using LTO-8, which is the current generation, you’d get a 10-pack of 12TB tapes for around $500. Noticeably cheaper per byte than hard drives.
I don’t recommend LTO-2 just because I don’t think that the drives are well-supported.
> While tapes are theoretically cool, the the drives are just too rare for them to be of any practical value to a home user. Even if the media has a better archival life than a hard disk (say 50 years), it won't do you a damn bit of good if there are no drives available in 50 years to read it.
If you’re evaluating just on the basis of the price of drives / media, then there’s a cutoff where tape becomes cheaper than hard drives. The easy way to calculate this cutoff is to divide the overhead cost (tape drive price) by the difference in the cost per TB of tapes and hard drives, which in the example here, is around $3k divided by $25/TB, or 120 TB.
In other words, if you have more than about 120 TB of data, then it is cheaper to buy a tape drive. I think any comments about whether tape drives are suitable for home use are really comments about whether you are interested in the use case of people who need to store >120 TB at home.
If you are running a YouTube channel as a hobby, or you run a side business doing videography for weddings, the tape drive starts to sound a lot better. The 120 TB cutoff might be around 300 hours of video, which might be only 50 events.
There are a lot of other reasons why you might NOT want to use tape, but it’s easy to have enough data that tape is the cheapest storage option. At “enterprise” scale the cost calculus is completely different and involves things like support contracts with Oracle (not necessary for hard drives), power/cooling in your DC (tape is very low-power), etc.
And let’s not forget that if you have 120TB of hard drives, you’re in the regime where you start having to buy multiple machines.
As for longetivity—I don’t have the data handy. If you are storing data on tape, you need to migrate to newer generations of tape, as old generations become obsolete and unavailable. If you are storing on hard disks, you also need to migrate because hard disks eventually fail (even if they are not powered on).
But on tape, the controller mechanism is separated from the storage medium. The controller mechanism is inside the drive readers itself, which will eventually fail.
So for both, it’s a trade off. They’re both going to eventually fail.
But it looks like the cheapest drive on Newegg for that is $3,299.00: https://www.newegg.com/p/2BM-0046-00001
> In other words, if you have more than about 120 TB of data, then it is cheaper to buy a tape drive. I think any comments about whether tape drives are suitable for home use are really comments about whether you are interested in the use case of people who need to store >120 TB at home.
> If you are running a YouTube channel as a hobby, or you run a side business doing videography for weddings, the tape drive starts to sound a lot better. The 120 TB cutoff might be around 300 hours of video, which might be only 50 events.
No dispute there, but I think most >120 TB home use cases run into the question of if it's even worth it to keep the data (raw/uncompressed). For most people, the answer is probably no, and curation/compression makes far more sense. For instance, I don't think it's even typical for wedding photographers/videographers to store their final product indefinitely, let alone the raw footage for a re-edit. You can keep a lot more events in a lot less space if you're only keep the raw footage for a few recent events and the 15-30 minute final edit for a year after it's finalized.
> And let’s not forget that if you have 120TB of hard drives, you’re in the regime where you start having to buy multiple machines.
Not if you swap disks as offline storage.
> As for longetivity—I don’t have the data handy. If you are storing data on tape, you need to migrate to newer generations of tape, as old generations become obsolete and unavailable. If you are storing on hard disks, you also need to migrate because hard disks eventually fail (even if they are not powered on).
Honestly, the theoretical longevity is the only aspect of tape that appeals to me, but like you said enterprise tapes will end up being like these enterprise optical disks in a fairly short period of time, so you're regularly going to have migration legwork to do or your data's effectively toast.
The nice thing about LTO as a format is its relative predictability and ease of acquiring the parts, even the very obsolete ones. It's all SCSI or SAS, most of the interesting stuff happens at the hardware level, with a bog-standard API. Your average backup app, whether it be Backup Exec or mtx/tar/etc. on Linux doesn't need to care about the media format. Unlike actual "enterprise" shops with datacenters and support contracts and such cruft, where the primary concern is "does it work", it is fine to buy older units second-hand. They are plentiful and cheap.
That's still... pretty terrible. If we compare you costs for a tape drive against a 14TB easystore @ $190[1], the break-even point is around 72TB. I don't know about you, but that's of data you have to store for it to be worth it. Even at 150TB you're only looking at around ~25% (~$500) in savings compared to hard drive, which I don't think is much when you factor in how much of a hassle tape drives are to work with.
[1] https://slickdeals.net/f/14587381-14tb-wd-easystore-external...
[2] https://www.wolframalpha.com/input/?i=solve+y+%3D+500+%2B+%2...
LTO5 onward supports LTFS, which exposes the tape to the OS as if it were any other removable storage, with the one proviso that deleting files doesn't reclaim space unless the entire tape is wiped.
It has been 8+ years since I played with LTFS, so maybe it has improved.
I was attempting only to answer the question and explain the source of confusion.
In native English speech, it would usually be expected that a business-case purchasor would speak of a more general case, perhaps by using "we".
“we” is even frequently employed by auctors writing papers to depersonalize.
Anyway, I guess this is the pedantry we (!) should expect from HN
The parent is clearly seeking to confirm serious long-term use, as you say, in considering the costs of failure cases.
Is this a question? I can't tell you if that's how you find my comment or not, sorry.
They wants a cheap tape device because they just want to use it in the home. Tape devices aren't cheap... because they aren't aimed at use just in the home.
They aren't aiming at home users - I think that's a simple fact not presuming anything.
Even if you believed they were a home user, they clearly demonstrated in their original comment that they did know that it was for long-term, high volume storage, and they did know the cost, and you appeared still to belittle them for not knowing.
It's a question in the sense of - why did you choose to say it in this way, which seemed impolite? What did you intend your comment to add?
Restore times are definitely long, but you can mostly avoid human labor by putting the tapes in a tape library, which will load tapes in the drives using robots. You still need technicians around because tapes / drives / robots will break, but individual restore operations can be completely automated.
But the real inflection point may come when 'all relevant information in your local disk'. The tapes here in question can maybe store every book every written!
We may be able to put the entirety of Wikipedia, every film, TV Show ever made, every book every lecture on a little disk.
The only thing we'd need to access in realtime would be contemporary data like traffic flows, weather situation etc..
The ability to store 'that much data quickly and easily' locally, may quite fundamentally change the equilibrium we have right now with the cloud and lead to a more natural decentralization.
"All of YouTube from 2008 until the present, 99 cents at the local gas station, on usb-like contraption"
So far, storage is still following something resembling Moores law, but it will probably hit physical limits way before a year of youtube (probably 30 of these new tapes or so) fits into your hand.
https://www.newegg.com/nimbus-data-dc-100tb/p/2U3-002M-00004
Following Moore's law of 2x density per 18 months, in 10 years we will see single drives of:
100 tb * 2^(10 years/1.5) = 10,159 TB
Meanwhile my phone's storage is basically now just a high speed cache for the internet.
Whether it will go back the other way again in the future, I don't know if anyone can know but tech does tend to be cyclical.
Now, I did it because I'm that type of guy. There's not that many people who actually do this bullshit, even though it's perfectly doable.
So why don't they? Because it doesn't make much sense, if you aren't afraid of upcoming nuclear winter. Wikipedia is updated and improved every day. You only sometimes want to refer to something old, but you nearly always want to check out something new. Petabytes of video are uploaded to Youtube every year. Probably TB/day wouldn't be an overestimation for audio on Spotify. All data is being updated constantly.
Also, the above is valid for pretty aggressive data compression. Is aggressively compressed data what we want? No. 2h video compressed into about 500 MB was totally fine 15 years ago. If I download a 2h movie today, it's normally around 20 GB. And by no means it's uncompressed.
Seriously, by now you should know for a fact, that if one believes there's such thing as "too much storage space" — he's stuck in the 80s.
And even if there would be such thing — realistically, a cluster of nodes in Google's datacenters can find you a book or a video you are looking for way faster than the most perfect HDD you could theoretically have locally. So, again, normal people wouldn't want to have all this stuff even if they could.
Remember when we could do 64Kb/s reliably over the web? Then you could do 'voice'. And after that threshold was crossed, you could basically do unlimited voice very quickly.
So Wikipedia - text only - is ballpark 50G - which is to say it would fit on a single mobile phone.
That is bigger than the first Google index!
I don't know how old you are, but the notion that you could walk around with this massive database, literally the size of all of Google right in their pockets in 1999 - would have blown people's minds. It was basically unthinkable.
The rate of growth of storage has slowed down a little in the last decade but there are still jumps to be had, and it's not inconceivable that we get 100-500TB storage in the nex1 10-15 years in regular devices, meaning 10-100x that in a slightly large storage device.
While video data is expanding (4K is much bigger than original HD) it can't go on forever, meaning, just like voice and text, once we cross a certain threshold, then it becomes irrelevantly small relative to storage as well.
So I think there's some value in my point:
In 10-20 years from now, as video storage becomes 'trivial' just as text is today (aka all of wiki text on your phone) - then huge amounts of data become available, instantly.
Though some data sources change a lot - others do not.
It's not inconceivable that we put the 'entire western canon' in everyone's homes.
The other thing no so evident in my comment is that there's only so much use for all of this data.
We are getting massively smaller marginal returns for all this 'big data' we store, frankly, I question in many cases if it's worth it at all. I think a lot of companies have been duped into saving every mouse click or whatever concerning every customer. The world is just not that complicated.
What this means - is as computers miniturize - and storage as well - we may see regular data centers shift away from the cloud back to 'on premise'.
If you can fit 'infinite computing power' in a little closet, and the parts are easily replaced ... then I can see companies doing that.
The promise of cloud computing today largely rests on the economies of scale of physicality: parts take up space, cables, power, heating/cooling, you want a lot of flex/headspace, configurations.
I don't see why in 20 years from no, you can buy 'off the shelf' a 'box' that has the equivalent of 100Ec2s, 500TB of storage and multiple G/s networking cards.
You could run an entire corporate office of 1000 people from just a single box.
The 'physicality' of it all would be mundane and irrelevant. Obviously, it would be 'super complex' and still need 200 IT people to admin all of the software, but physically it could be small and cheap.
'Big maybe' of course, but there are some possibilities in there I think.
https://www.theatlantic.com/technology/archive/2017/04/the-t...
IBM marketing smort
What I really want is an affordable Tape storage for us common folks / home users to archive data long term. It's really a pain to keep transferring your old backups every 2-5 years to a new CD / DVD / portable HDD. There is a market for home users that really don't want to put their data on the "cloud". And this generation is really creating so much data, some of which, I am sure they would like to store long-term.
You're going to use the drive for 50 years? 50 years? Half a century? And you think it'll be working and supporting a format with a useful capacity in half a century?
If anything, ZFS makes that whole system you just described easier and more automatic. Its not really hard for me to push the "scrub" button on my Nas4Free box every few months.
OK, but OP said:
>It's really a pain to keep transferring your old backups every 2-5 years...
With a new HD you can just copy it across in one step.
rsync --archive --acls --xattrs --checksum --vvv /src /destIf the OP really needs dozens of TBs of capacity every few year, they definitely don't fit in the home user market they are talking about.
Money - The cost factor of buying the new HDD. And to be safe, you need to have more than one medium. That's again more cost.
Time - Ensuring that your existing machine is compatible with both your old and new disks (IDE vs SATA or working DVD drives for example).
I know your comment was a throwaway idea but I got a little bit too involved looking up information around this answer - so I've committed further and looked a little deeper to avoid looking too stupid: Currently with BaFe (Barium Ferrite) as their magnetic particle, LTO-7+ tapes have particle sizes less than 100nm in size (https://indico.cern.ch/event/637013/contributions/2669089/ - check out slides 14 and 15 in the powerpoint.).
VHS on the other hand is a little more tricky to find information on I suspect due to the age of the technologies involved it's going to be significantly bigger - but the closest I've found to a technical document is a student paper at NYU which refers to this IEEE Paper (https://ieeexplore.ieee.org/document/50474) saying that the average particle size for VHS is ~300nm and for S-VHS at ~150nm.
But that first presentation also mentions smoothness increases over time with LTO, which suggests improvements in the coating process.
I suspect improvements in the coating, combined with both the size of magnetic particle shrinking, enabled an increase in density of particles on the tape. This in conjunction with improved thinness of the tapes enabled large increases in tape length for the same cartridge size (nearly double the length from LTO-1 -> LTO-8), has in turn led to the enormous jumps in capacity we've seen in LTO.
Meaning unfortunately I don't think it'll be possible to do super interesting things with VHS.
I don't recommend backing up to HDD's. They are prone to early failure from portable use because of vibration and drops.
What we need are more affordable sata SSD's. Currently nvme SSD's are very close to the same price.
If you could sell a 1TB portable usb 3/usb c backup device that comes with good software the vast majority of people would be set.
Your backup device should stay at home, on the desk or in the closet where you hide your networking equipment. There's very little benefit to trying to carry around your backup device with your laptop on a regular basis, and if you do, you still need another backup device that isn't going to get stolen at the same time as your laptop.
This. Even 100MB/s SSD will do. We need a type of NAND and SSD that offers High Capacity and low speed and low cost. The Current 2TB is ~$170, compared to a 2TB Portable HDD at ~$60.
And I am wondering, if Silent File Corruption, bitrot etc are things of the past on SSD? Does BTRFS / ZFS even make sense on SSD?
[1] https://www.amazon.com/Green-1TB-Internal-SSD-WDS100T2G0A/dp...
They say that they have a 50 year lifespan, but everyone in the industry knows that's a blatant lie these days. The tape customers transfer everything over to the new generation every 2 to 3 years, so it's never tested and none of the customers really care.
You also only get a couple hundred writes to a tape in it's lifetime. That includes bulk appends as the stress from seeking is a major factor of what kills them. They really need to be treated as WORO media, bulk, the whole 12TB or what have you at once. Not a great fit for home use.
These days the main benefit to tape is the physical safety of having the media be inside a cartridge, and the streaming read speeds are stupid fast so you can reconstruct a backup faster.
Home use, even with a data hoarder sized NAS, is better served with a bluray burner with multiple copies.
Again, tapes are for archival storage. Not nearline, not live, archival. As in, you write the data and likely don't come back to it for months to years.
These tape-drives have 300MB/s+ of read/write speed. Sequential yes, but almost all backup tasks are sequential.
Optical has 10MB/s or so. You get more out of a 100Mbps connection to (insert cloud storage provider here), let alone Gbps.
At a minimum, a modern, reasonable mechanism for backups needs to be faster than cloud (100Mbps or Gbps), otherwise its basically worthless to the consumer. Hard drives and Tape get there, but there was no real way to improve Optical's read/write performance (outside of overengineered "jukebox" robots available to Facebook and a few other select groups), so it went fully obsolete.
I just gave a look at the first Blu Ray media on Newegg.com. Here's what is search #1: https://www.newegg.com/verbatim-6x-25gb-bd-r/p/N82E168171301...
That's 6x speed, or roughly 25MB/s. You can store 25GBs per disk and it is going to take 1000% longer than a $150 5TB Hard Drive AND you're going to have to sit there and remove / add a new disk every few minutes.
And for what? The hard drive is cheaper after all of that.
And yeah, you'd be buying 16x 128GB media in this use case. Not the bottom of the barrel stuff for giving home movies to grandma.
So push the "ZFS Scrub" button every 6 months.
Don't store hard drives. Store a NAS. The entire freaking computer is stored as a unit. Every few months, turn it on, push "ZFS Scrub" to double-check the data, and you're set.
BluRays also degenerate over time unless stored in proper UV-sealed conditions / temperature controlled conditions. Everything requires a degree of maintenance and checking. The question is how to best automate that checking process.
Hard drives are read/written to at 200+ MB/s, making these maintenance checks much faster. They're also bigger, which means no need to manually insert / remove disks from drives. This entire process could be automated from a "Wake on LAN" packet and a few clicks from any terminal (phone, computer, whatever).
-------
I dunno. I'm looking up these BD-R XL drives you're talking about: its like $30 per 100GB disk or something. That's an obscene price for so little storage. I guess if all your data fits in one of those disks its fine, but... I got some archived movies and stuff on my NAS (50GB per Blu Ray). Needed to store some of my video editing files (so I need the original data that matches with my video editing files).
The real data of importance of mine probably fits in a BluRay. But then I won't have all the other stuff I've saved up "because I can" on my 2x 5TB NAS (Mirrored, so only 5TB of storage).
3 copies, 2 different media, 1 offsite.
I've seen large orgs do what you're saying and lose nearly everything because one bug took them out completely.
Ehh? Not really. All of these things we're talking about are components of our backup strategy.
If I wanted an offsite backup, its just rsync to some cloud provider (like rsync.net) or something. I don't do that because I don't think my data is worth the recurring cost of that, but its an option.
-------
My point is that when my solution is 200MB/s and on the order of $500 to $1000 for the component ($500 if you build your own NAS, $1000 if you buy premade parts. Assuming 2x 5TB hard drive is just $300 for 5TBs of _mirrored_ storage)...
While your proposed component costs $30 per 0.1TB and read/writes at a lousy 70MB/s... to get the equivalent mirrored setup you need to buy 100 Blu-Rays or roughly $3000 in Blu Ray XLs alone (2 copies of your data across 2-different Blu Rays, for the same redundancy factor as the 2x Hard Drive solution).
So I have to raise my eyebrows a little bit. How are you checking that the data doesn't degrade? Are you manually checking all of those BluRays you've created for reliability? That's a lot of sitting around and inserting / removing drives.
Its literally cheaper to build a 2nd NAS and stick more hard drives into it, and keep that 2nd NAS offsite somewhere.
-------
If you're gonna hold my feet to the fire over this "2-media" thing, then my 2nd media of choice would be Flash storage before Optical. Because SPEED is king. Speed means you can checksum your data and ensure that your backups are still good. I think Flash is a bit expensive compared to HDD, but based off of these BL-XL disks, I'm thinking that Flash is actually cheaper than Blu Ray and something like 10,000% faster. (Tapes would be a more ideal 2nd media... but I'm not "big enough" to make tapes cost-effective).
Yes, checksums and scrubbing. If you want to protect against bitrot (in ANY medium), you MUST double-check your backups over time.
TEST your backups. Any backup strategy that doesn't have a regular testing schedule is null-and-void in my opinion.
I seem to think that HDD, then Flash, and MAYBE Tape (if you're going really, really big) are the mediums of choice of the modern computer user. I'm not really seeing where Optical fits in today's world. Maybe a future Disk format (with maybe TB-level disks) can make Optical a thing again... but 100GB Blu Rays aren't really in the works for the modern user.
70MB/s is slow. And 100GB is too small per storage unit.
Objections on the failure rate of HDDs are absolutely valid, but then one should also need to consider the bigger picture (e.g. storage loss), in which case, having a remote copy is also important.
You have an interesting argument - by the same reasoning, when one has eventually filled up a 12 TB HDD, it would no longer be hoarding by tomorrow's standard. In other words, at this point, one should be able to get the next generation of HDDs for cheap, and it's this fact that makes all tape drives unnecessary.
Now I wonder whether it was a mistake for me to buy a spare 14 TB HDD to 1:1 mirror my new file server for cold backup. Perhaps a smaller one should be good for 5 years anyway...
Indeed. The home directory on my PC at home has 80GB of files stored. Most of it is just there out of laziness. About the only thing I try to keep backups of are my tax returns.
What do people have on a home machine that needs enterprise-grade backups?
You might say photos and videos. I guess that's a personal thing; I realized a long time ago that I never spent any time actually looking at any of the pictures I took, so I stopped taking them.
I realized in the last year or so that the only digital media I have which I would be genuinely sad to lose were wedding photos - so I saved those to 3 different cloud providers, made a Blu-ray copy, and a copy on the NAS. If I lose all of that in some tragedy, chances are I've got bigger things to worry about.
That is a completely logical take. However, I sort of think of it as a posterity move. Although my kids might not particularly want to see pictures of themselves, their great grand children—or beyond—might love it. I would pay a pretty penny to see home movies of my grandparents, whom I never knew.
It adds up quite fast. And being able to have put everything on a tape every year with a label on it would give me some peace of mind, even more if I place 2 of them on different location.
You're basing your argument on extreme and unrepresentative assumptions.
First of all, you do not need to fill a storage device to the brim to justify it's use. You can use the exact same rationale to claim no one needs 500GB HDs even though they are pretty much standard these days.
Additionally, you falsely assume that your data storage needs start the very moment someone buys a drive, and up till then they have no data lying around. That's not the case at all. People buy high-density storage devices because they already have the data lying around, and they don't want to lose it. You're ignoring that people already have piles of CDRs/DVDs/Blu-ray disks lying around.
Additionally, you are somehow assuming that people would buy storage devices for their density, ignoring their primary use case: long term data preservation.
I cheerfully quantify just about everything in my life, yet somehow I missed that one. I am a little bit of a data hoarder, or maybe just a little paranoid. Seeing those numbers is actually quite helpful. Thank you.
PS I get downvoted a lot for comments like this, so in case it sounds sarcastic or facetious, it is not. I mean it sincerely.
I think the market for tape will always be the same few giant companies for whom it will be priced just a bit less than the next cheapest option.
The problem was, nobody actually backs up so not enough people bought tapes/drives so they quit making them.
During the 80s there was no digital consumer camera market. Nowadays every person can easily generate hundreds of megabytes of photos and videos per day. Each hour of a 4K video can be close to 7GB, and we're already seeing cellphones which are able to record 4K video at 60fps and 1080p videos at 240fps.
On the Mac side, in 1989 you could apparently buy a Mac IIci for $6200 with 1MB or 4MB, "expandable to 128MB".
Some of the issues you need to address:
- Trust. How can someone trust you with potentially proprietary / copyrighted data?
- Speed. Is it faster to copy the data to a tape locally, then to transfer to you?
- Reliability. How can the client be sure that you have made a successful backup?
- Advertising. Did you reach your target base to make them aware of your product and address all their concern to make them consider it?
Trust: yep, hard to say for this one. At least I can provide actual guarantees (for instance, none of our storage is out of the country).
Speed: it's possible to send us tapes directly. However the main selling point of our solution is to allow people to upload relatively small volumes of data continuously, then archive it all to tape in large batches when you have enough to fill an LTO.
Reliability: the interface allows the user to see where their files are (on which tape), to run checksum controls on tapes, and restore from tape. Every tape is sold with 3 free operations a year (archive, checksum verification, restoration).
Advertising: that's clearly not our strong point :)
If you don't do that, eventually you'll get to a point where it can't repair anything, and then you gain nothing over using a filesystem that doesn't do this, is the point.
So unless this is an incredibly radical breakthrough, that’s the timeframe I’d expect for the headline to become a real product.
Note that the article shows a table from IBM claiming they achieved 35TB-per-cartridge capacity in 2010, and that still isn’t something you can buy.
Now that GDPR applies, most companies need to rewrite backups every 30 days anyway to remove data where a GDPR deletion request applies. That tips the scale further in the direction of always spinning hard drives. Just hook up 64 drives to each machine, make sure you only do streaming writes of 1Gb+ files, do some ZFS raid-like scheme, and away you go.
It was mostly because a user might share data with another user, for example two users at the same postal address. Our fraud team needs to be able to look stuff like that up, so there need to be database keys on stuff like that, both in the backups and in production. If one of the users with a specific address has a GDPR deletion, we need to delete that users data, but if another user has the same address, we still need to keep the address itself. Yet if both users have a GDPR deletion apply to them, as well as deleting the address, we need to delete the fact both deleted users had the same address (even if we don't know the address, because the pattern of which deleted users shared info with other deleted users could identify them)
See... it's complex! Simplest solution properly delete the data and rewrite the backups!
How thick would that tape need to be to allow the motorized spool to pull the tape from the other (passive) spool?
Imagine you were to make an assembly that could hold two spools of this tape, how large would it be? It might be hard to fit it through the door of your server room. And it might not even fit in the van that carries away your off-site backups :)
(Then I had to look it up. Henry IV Part I, Act III, Glendower: I can call spirits from the vasty deep; Hotspur: Why so can I, and so can any man, but will they come when you do call them?)
First of all about the security of this system. Yes, it is very avantgarde but being a physical support I will always have the worry that it can be damaged.
If I immagine to use it to store sensitive information this worry rise up. It can easily be accessed or stolen or damaged.
And what about the possible cost of a solution like this?
a part from this aspect it is incredible what Fujifilm has been able to reach in terms of tech
I do not find this to be all that impressive, frankness be.