Where did you get the gear?
How cost-effective is the overall setup compared to spinning rust?
How is driver support in popular operating systems?
Is there more you'd recommend taking into consideration or learning before jumping in? Especially beyond the obvious realities of extremely high latency, of course ;)
I never ended up using it personally, but when I was considering it, used gear on eBay was top of the list.
> How cost-effective is the overall setup compared to spinning rust?
For personal use, it is not even remotely cost-effective unless you are a data hoarder. The cost of the drive is simply too high. Also keep in mind that the drive can become damaged or need servicing.
It only makes sense if you are dealing with e.g. large quantities of video. The article cites a minimum 140TB where tape becomes cheaper but this involves some dubious decision-making.
> How is driver support in popular operating systems?
I have only ever used tape with Linux + custom software so I can’t speak of software support in general.
> Is there more you'd recommend taking into consideration or learning before jumping in? Especially beyond the obvious realities of extremely high latency, of course ;)
The obvious reality to me here is that it just doesn’t make sense unless you’re in spitting distance of a petabyte. The only thing I can imagine someone doing personally is video, and it would have to be a lot of video like some kind of long-running documentary project that you are producing in your spare time.
The other factor is to be more paranoid about mixing tape generations than the vendors suggest. Ignore the specs and stick to same-generation media and drives.
Efficiently writing to a tape means filling e.g. a 20 MB/s write channel for LTO-6, or 750 MB/s for LTO-8. Can you sustain 750 MB/s? It’s not so easy in practice. You throw away tape capacity if you get buffer underruns.
The drive doesn't pause while the buffer fills again? Considering the price of the drives, I find it very surprising that they don't do that.
The drive does stop, and it runs at different speeds depending on how fast you are feeding it data. However, this process wastes tape capacity as you switch speeds.
Do 10% of a tape at a time, buffered on a $200-$300 NVMe SSD?
It's cheaper to run the tape slower (you can adjust it so that it is very close to what you can sustainably deliver)
Not meaningfully, unless there's something ridiculous about the format I'm unaware of. A full tape write covers 160-200 linear kilometers. Stopping 10 times, which is once per hour, is barely anything. I haven't found anything listing the exact size of gaps, but I did find a post from someone having thousands of buffer underruns that was only losing half a second of capacity per underrun.
This means that each 50 GB file is written at 3 Gb/s, the maximum tape speed, but after each 50 GB the tape is stopped then restarted. So I write on each tape about 120 files with an aggregate size of up to 45 905 860 blocks of 131 072 bytes each. That is more than the 6.0 TB advertised for the tapes, so I do not waste any measurable tape capacity. The 50 GB file size is chosen because it fits inside the 64 GB RAM of the computer with the tape drive and certain algorithms that I am using to process the tape files (lrzip & par2, for compression & redundancy) are faster when the entire file fits in the RAM.
The drives are bloody expensive. Mine was around $5k.
LTO8 tapes are still hard to come by even though the patent dispute is settled. This should change soon.
The software to archive to tape is clunky and frustrating but it works. I use something free that integrates with Finder on Mac. I'm away from my office so dont remember what it's called.
The main benefit is that the tapes last for half a century and are very cheap per TB once you have made the initial investment in the drive. Very good for long term archival of films and source footage which is what I use it for.
When you do a lot of those, drive price isn’t all that expensive. My major gripe with it is that you couldn’t hook it up to a USB3. Rare ones had thunderbolt, but most were that bloody external SAS thing. Not really all that great when you don’t have those controllers around. Beats the purpose of external drives.
https://magstor.com/collections/magstor-thunderbolt-3-tape-d...
I use self-system engineered (read: self OS packaged) Bacula software for backups with a custom developed shell program which interfaces with Oracle RMAN to do backups.
I use Solaris 10 on sparc, which also goes for pennies on the dollar nowadays since there are few of us left who understand how to run it and even fewer who understand how to squeeze the most out of that hardware + software combination.
The tape library shows up as character and block files under /dev/rmt/ and the robotic arm with the barcode scanner under /dev/changer/, nothing special or fancy about it. There is another piece of freeware I compiled and packaged (whose name I forgot) which controls the robot arm and the scanner, also by issuing standard SCSI commands to /dev/changer/ files, nothing fancy or special about it. The tape library has enough capacity for full backups with three month retention policy, so the only thing I have to do is replace a worn out tape cartridge roughly once in three years. Backups are daily. The entire solution ran me around $800 (server + differential SCSI controller + tape library) and the tapes ran me $500, over the span of 12 years.
I have an old Quantum DLT S4 drive and something like 18 tapes. Each tape is 800GB uncompressed and I get about 1.8x compression on average. It's an external unit connected via SCSI ultra 320. The ultra 320 SCSI card is something I picked up from ebay for like $20. The drive and tapes I got from a former consulting client who was going to trash it.
It was the last of the DLT line and is similar to LTO4. Get an LTO5 if possible; it has some cool features like partitions.
I only store my most critical stuff on tapes. Each weekend I do a tape swap and take the old tape to work on Monday and bring the oldest one back. I keep the tapes at work in case my house burns down or something (I lived through two house fires as a kid so I'm super neurotic about it, don't ask). The tapes are encrypted so nobody can recover from them if they get one. If my house did burn down I'd have to somehow procure another tape drive, but that would be the least of my problems.
I just use a custom tar script to do it. I tried both Bacula and Amanda in the past but they were too complicated or had other issues. My own tar script did what I wanted.
> How cost-effective is the overall setup compared to spinning rust?
Since I got most of it for free and I already had experience with tape, it was pretty good. I've had it since 2008 so it's over ten years old and still going. I also have some UBS3 backup disks that I use for daily backups. In all this time, only one tape has died and I do a full validation of the previous backups every time I bring a tape back and re-use it.
For new installs, go with some USB3 disks from Seagate or WD -- both are great, I have both. Avoid Toshiba due to some weaknesses in handling bumps and vibration during runtime due to firmware (Toshiba internal drives are great though) -- Don't ask me how I know this. Seal them into some kind of fire-proof/flood-proof safe and store somewhere that a fire/flood won't destroy it.
> How is driver support in popular operating systems?
I'm all linux. It works. All that matters is the SCSI card being supported. Make sure NOT to get a RAID card, regular SCSI only. The RAID cards often won't work with tape drives because the firmware is made for RAID drives, not tape.
> Is there more you'd recommend taking into consideration or learning before jumping in? Especially beyond the obvious realities of extremely high latency, of course ;)
So the only real advantage for me using tapes is that I got most of it for free, the tapes are more portable and lighter than a USB hard drive for carrying around. Almost everything else is a negative compared to USB hard drives.
I use some UBS3 hard drives for daily backups via an rsync --link-dest script that de-duplicates everything and that works great. For $100 you can get a 4-6GB USB hard drive from WD or Seagate and for my important stuff, that works fine.
I'm about to fill up my older 5GB drive -- it's down to about 200GB free space left. I already have a new 6TB drive I brought recently and when it gets down a little lower, I'll just replace it, do full backups, and put the old drive into storage for permanent archival.
Data backups is a full occupational specialty, like being a DBA, network engineer, security, or web monkey. I did TSM for awhile. There's a ton of issues in the hardware, strategy, security, performance, and more.
Do yourself a favor and just set up something simple that JUST WORKS and don't fuck with it too much.
Do a practice recovery once and that'll teach you how good your setup is and improve.
We ( work ) have a massive pile of DLT, mainly S4's...these were written 2005 to 2009 ish and they still restore just great. I mean, i wish they wouldn't as i'd love to be shot of them but they are too reliable.
The casual observer won't understand the difference between "ability to restore" and "doing backups", but it matters when the building burns down.
node_modules is shit anyway, and shouldn't be duplicated for every single project.
I have yet to find a good solution. My phone is crippled by the library and none of the Macs like it much either. I’d guess I was at 600gb.
I'm trying to use icloud photos, because then I have them where they belong on my iPhone, but I just don't like the way it works. It's slow on my mac, but it's nice to have thumbnails locally, and originals available when needed. Althought that also doesn't work nicely when dragging. It's cool that photos groups things on event, but for me, I like to have my "camera roll", and manually make albums. That's possible, but it's ust easier to drag files around. Also, I'm looking for a solution to 'archive' photos (hide them, because I took 10 pictures of 1 thing).
There was a time I wanted to write an app/service for this. I might still do that if there's enough demand.
I also haven't figured out how to archive some video material. Recently I started shooting small clips, with the intention to make small 5 minute videos of my trips. As I don't want to delete the original material, I also need a place to store them, but I don't want to "browse" them.
Pre-tape, I used disks, rotated them and kept them in a safe. Burned optical discs of my most important stuff. I had two experiences that caused me to change: I had a failure and went to do a recovery and in either haste or anxiety (I put this on me, I should have paused and took more time but I was rushing) I knocked the most recent drive off my desk and effectively killed it. The other was a bit more subtle (oddly, I have only had 3 backup recovery situations and I messed up 2 of them) but my “backup” was rsync to disk on cron and a backup started during restore and began to “remove” my good backup because the master didn’t have that restored yet. There are a lot of lessons in there, most significantly, “backup” is a different thing than live access to data, you want it versioned and somewhat fixed. Rsync isnt it. You don’t want to accidentally delete your backup. Tape switches your mindset on that stuff. I’m not going to pretend it’s cost effective for my needs (although I have a lot of hd+ video of my kids) but it’s durable, it’s purpose built and cold storage can’t really be hacked. There’s no monthly fees, no risk of a cloud provider sunsetting my tier of service, and 25T of tape is like $250. Drive MTBF numbers are big enough to be abstract and yet you will likely encounter it at some point, tape has these practical numbers that are ultra conservative and come from decades of experience; you sort of keep it in mind when you’ve used a tape a dozen times or so. That being said, when the drive flakes, I’m in for an expensive upgrade/replacement and when I do upgrade, I should copy my tapes to new ones with the new format which will take time and money. With SMR, it’s almost like discs are taking on tape like qualities and it’s not hard to find stories about people getting crap performance on their new disks. It’s not sexy but I feel very confident that I’ll be able to get my data if I need it. It’s really easy to store a tape in your safe deposit box or at your parents house too, they won’t accidentally kill it by knocking it off a shelf.
The tape cost is about 10 EUR per terabyte. Previously I was using HDDs for archiving. I have not followed closely the evolution of the HDD price, but looking now on Amazon I see about 30 EUR per terabyte.
I have a data archive of about 120 TB, but I am paranoid because of past data losses due to defective HDDs/CDs, so I store 3 copies of each tape (in different places).
That means that by using tape instead of HDDs, I have already saved 20 * 360 - 3000 = 4200 EUR.
Someone who would store just 2 copies of each tape (the minimum rational value), would need to have a data archive of at least 75 TB to recover the initial cost.
Nevertheless saving money is not everything, you also gain peace-of-mind because the chances of the tape becoming defective just from storing are negligible compared to a HDD, which if not used for a very long time might never start spinning again.
With tapes you must transfer the data only when you can no longer buy drives that would read them. With HDDs you should better replace them before their warranty expires, as I have seen too many that died immediately after that.
The tapes work OK both in Linux and in FreeBSD, but the FreeBSD utilities are more convenient, so I have connected the tape drive to a FreeBSD server and I transfer the data to be archived through 10 Gb/s Ethernet, which is faster than the 6 Gb/s SAS, so it is not a bottleneck. The writing speed of LTO-7 is 3 Gb/s, so that is the bottleneck anyway, not the interfaces.
That speed is much faster than of any HDD of up to 8 TB that I have ever used, but I have not checked the latest HDD models to see if they have improved their speed to levels comparable to tapes.
Using tape is very simple and retrieving any file (I have a tape index on my SSDs) takes at most 3 minutes, including the time to manually start the drive and inserting a tape in the drive. The maximum seek time is about 2 minutes. When I had a HDD archive, there was still the same time for manually selecting a HDD and inserting it in the HDD rack. For retrieving very large files, the tape can even be faster, due to the faster sequential speed.
The conclusion is that if you have data of at least several tens of terabytes and/or if you value that data enough to want to avoid any chance of losing it, than magnetic tape remains the best solution for now.
I’ve only ever read bad things about mixing generations despite the fact that the drives are capable of this (eg LTO5 tape in LTO6/7 drives) though I’ve not done this personally - but I wouldn’t recommend doing so based on this. Don’t be tempted to buy really old generations as if you ever need to restore in a disaster where you only have the tapes, you then have the nightmare of sourcing a new old drive too which by that point might be a lot harder.
You’ll want to think carefully about software setup. You’re probably going to wind up using some form of custom scripts unless you can put in the time to set up bacula properly and are sure it works for you (bare metal restore can be a tricky aspect of any system to verify it works properly). If you script your own then make sure you also understand the drawbacks of tar. tar as a tool is highly compatible which is good but is poor at recording incremental backups (there are some custom extensions which gnu tar has but this reduces compatibility) if you want to go with a more complex schedule than full backup every time. Tar also makes it difficult to seek to just the part of the archive that you need to get a single file. Again there are gnu extensions for this but this isn’t the best solution. Finally, tar then gzip means if your gzip stream gets corrupted then you might lose quite a bit of the tar archive so it’s not very resilient. You could rely on the drive compression but you might also consider other solutions, especially ones which also include checksumming. Consider either scripting around these or better what other tools could take its place (eg I have used dar in some of my setup as it handles this better but then you need to think about keeping a working binary around which can unpack your archives - I put a static compiled binary in my archive sets just in case).
Finally if you go down the tape route then consider strongly spooling your stream which will be written out to tape onto disk first, to give yourself maximum chance of getting the more consistent throughput onto tape which has benefits in terms of tape reliability (and may indeed wind up being quicker overall). That also allows you to easily write multiple copies of the same tape if you intend to send one or more offsite and keep one locally.
Where this setup works well for me is that when you start wanting many copies of your backups (offsite, multiple point in time full backups etc) then the cost effectiveness becomes a lot better than the equivalent with disk. I also trust tape to survive longer term on a shelf than disk.
Since I don't have enough karma to write another comment anonymously:
Restoring from backups is a last resort for very important files deleted more than a few days ago (beyond what hourly/daily snapshots contain) or for business continuity (BCP) / disaster recovery (DR) of an entire system that cannot be recovered via automated configuration management rebuild. Snapshots to nearline storage reduce RTO for most common use-cases, and most users and systems don't generate much churn of data in a day. For recovery of important files, sending a courier out to retrieve a backup tape set from the vaulting provider implies the RTO would accept in a matter of several hours. An RTO of under an hour would imply backups would need to be stored on nearline media (commodity spinning rust) or at a backup cloud vendor like AWS Glacier or Tarsnap, which is more expensive than tape.
The deciding factor of how much to rationally spend on DR/BCP solutions follows from a Business Impact Analysis (BIA) to quantify the loss of a service over a certain time, which drives RTO and RPO goals. (A common factoid of DR/BCP is "about half of businesses who lose their data go out of business within six months.")
RTO - Recovery Time Objective - desire/need to recover within a certain time window
RPO - Recovery Point Objective - how close to the original is the recovery