Hacker News Reader

Comment by dragontamer | Hacker News Reader

Assuming you can read/write at 512 MB/s, it would take 24-days to read/write a full Petabyte.

Despite the long time to read/write, any situation where capacity is king (aka: any Tape-drive solution today), probably benefits from something like this.

This isn't a solution where you backup / restore from per se, but instead provide dozens of backups to (and probably only restore once from). Every day, you backup up to 42TBs of data (512MBps going full tilt for all 24-hours).

Then, when your backups fail, or someone asks for data from 3 weeks ago, you rewind the tape and grab that data. Crazy that this would all fit on one tape potentially.

View on HN

> Assuming you can read/write at 512 MB/s

I realize that's just an assumption for the sake of discussion (and thank you for labeling it as one), but I think it's a little on the low side.

Wikipedia says (https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations) that LTO-9 can write at 400 MB/s native and LTO-10 is going to be 1100 MB/s.

Since this new Fujifilm format is denser, I would expect it to be even higher speed. Although speed and density don't have a simple linear relationship, speed generally increases with density.

Generating enough data to keep the tape moving without a buffer getting empty is an important consideration, but I think this product is mostly aimed at people who have enough data to back up that they can fill a tape in a few hours. (Otherwise, why spend the money on cutting edge technology?) So they probably have lots and lots of machines to back up, and you can stream backups in parallel, writing to a staging area and/or multiplexing onto the tape.

Also, there's some risk if you leave a tape in the drive and append to it over the course of several days. While it's still in the drive, it's not quite an offline backup. The drive could malfunction and chew up the tape, or some software could accidentally rewind and start overwriting.

I decided to invest in an LTE drive to periodically archive my ZFS based NAS. This relatively old drive support 50-150 MB/s with a small internal buffer. My biggest problem ended up being that the default tar util on ZFS on spinning rust cannot even saturate the 50MB/s consistently, so the drive had so stop and rewind (called shoe-shining). It would take ages to backup parts of my filesystem with lot of small files (i.e. git repos).

I ended up writing a custom software with an adaptive scheduler to avoid sudden speed changes.

That is to say as far as I am concerned my ancient LTO6 drive is too fast...

It's been best practice for a while in the 'enterprise' realm to do disk-to-disk backup first so that the streams are all sequential and then copy to tape. Enterprise-y software can also generally multiplex multiple clients to a single tape.

I used Amanda Backup[0] for years on a Unix farm, and that's exactly how it works. Client apps on all of the machines being backed up ran tar and streamed the output to a backup server where it was written to disk. Then the completed files were streamed out to the tape.

[0] http://www.amanda.org

I don’t understand how this scales. If you have 1 PT to backup to tape, your first have to write it to disks on a backup server? So you have to buy 1PT worth of backup disks first, is that correct?

You wouldn't usually backup the whole thing first. That 1 PB is probably a bunch of smaller projects, so you'd create a tar of the first project, then flush it to tape. Tar up the second project, then flush it to tape.

It's not so bad if you're writing 23 tarballs out to tape and pausing between each one. What you really want to avoid is where the tape drive is pausing many times during a single archive, perhaps because tar can't keep the drive fed while it's traversing a big directory of teensy files.

You'd never buy a 1PB tape drive if you only had 1PB of data to backup.

A 1 PB (1000TB) tape drive would probably cost $10,000+ to $100,000. The tapes themselves would be cheap (cheaper than hard drives or SSDs). So you'd buy many tapes, perhaps 50 of them, to be cost-effective.

That's why a lot of comments around here are talking about tape libraries (entire boxes of tapes). You must plan to use more than 50 tapes (aka: 50PBs) before you reach any level of cost-effectiveness.

---------

From there, we see that 1PB of data is simply "infrastructure", to help you lay out the data before it gets to the tape. There are 20TB hard drives today: a single machine with 50 x 20TB hard drives (and maybe some flash storage to accelerate the write to hard drives) is what we're looking at to feed the Tape Drive.

When I was a kid and we had to do a school internship, I went to my dad's company. Since we were just school kids, they didn't really give us anything to do. Except: Loading and unloading tapes and moving tape lists from the main tape library room to the backup safe. The backup safe was several hundred meters away in a different building (on the same lot).

This was basically their WORM archival system. The tape library was a room full of tapes, wall to wall with shelves in the middle. Those were the oldest tapes. The in-use tapes were inside "Robbie", the circular tape robot. Not sure it it's this model exactly, but it sure looked like this: https://bjgreenberg.files.wordpress.com/2012/05/tape-library...

The tape robot would periodically spit out tapes on one end that we'd collect and then file away in the correct location on the shelves and vice versa, got a list of tapes to get from the shelves and insert into "Robbie". I don't remember typical tape sizes at the time, except that obviously it was much more capacity in a little tape than I had in my hard drives at home at the time :) Now imagine, Robbie filled to the brim with these 1PB tapes...

Robbie and his extended tape library was in a room right adjacent to the actual data center floor, which had an IBM 390 mainframe and lots and lots of EMC^2s, AIXs, Suns and everything else you can think of being in a data center ca. 2000. You obviously had to get into and out of the data center through a single person entrance gate (a vertical tube that literally only one person could fit in, opening one side at a time only). First day I'd just walk through the data center and look at everything, reading labels and such. Found the server that all my personal emails were going through, as my ISP was hosting there! Heaven for a kid like me.

Sure, why not? It's an incremental cost of the backup system. Look at the current generation stuff. An LTO-8 drive puts 30TB onto a tape, but that's assuming 2.5:1 compression, so the native capacity is only 12TB.

Let's assume you're actually going to do the compression as you build the virtual tape. What's a 12TB hard disk cost these days? A lot less than the $4,000 that the tape drive costs. Heck, 12TB of SSD is going to be cheaper.

60x18TB drives and a server to put them in would cost about US$ 45K:

* https://www.45drives.com/products/storinator-xl60-configurat...

That gets you about 1PB raw.

I would highly recommend jacking up the --blocking-factor on tar (or your custom software) as high as your drive will accept. AFAIK the default on tar is 512b which is utterly inadequate, and even sequential data on a fast SSD will shoe-shine when written directly to tape.

AFAIK most large businesses wind up writing to disk and then queuing the resulting serialized virtual tapes for actual writing at a later time. This basically turns the virtual tape library into a massive write buffer. It's absolutely insane that we have to do this - but a lot of archival utilities aren't built for async/parallel IO and thus aren't taking advantage of modern storage devices.

The tool I ended up writing does essentially something very similar:

- It uses a block size that matches the disk

- It uses 50% of my RAM as a ring buffer

- On start (and on buffer underrun) it waits for the buffer to fill before it starts writing to tape.

- It adjust the tape write speed as an exponential function of the buffer fill level. I am proud of this last one. It is very simple and automatically converges to the best average speed to keep both the tape speed and the buffer level relatively stable.

I have been considering making the reading multi threaded to make use of more disks, but that would be a bit of an overkill for my personal 2 disk NAS.

edit:formatting

- How many TB are you backing up?

- How frequently do you perform backups?

- How often do you do fulls vs incremental?

- What drive/autoloader are you using, and what was the total investment in drive + tapes?

I keep considering a used LTO but I've only got 20 TB and it doesn't seem worth the trouble. I should just buy another 20 TB of drives and keep them offline when not backing up.

I back up around 20TB and the easiest solution I found was to just buy 4x12TB drives and mirror to cloud (yielding 2x offline copies). For stuff that's really really important, around half of that, and which if restore is needed fast access is required, I also back it up to SSD.

It's not the greatest solution, but when you're hovering at some small multiple of common single drives, it's certainly the easy answer.

I only have ~1.5TB stuff that is really important which fits on a single LTO5 tape. (I said LTO6 before, I was mistaken) I archive every ~3 months, generally before I make bigger changes. I always do a full tar archive, it typically takes the whole day, definitely an event my family is aware of. I use a desktop HP Ultrium LTO5 drive. I fit on a single tape, but even if I didn't, changing tape twice a day would not be an big ask.

Months of researching, ~$900 total on 2 used drives (first one was broken) + enclosure + SAS controller + 5 tapes.

You need the enclosure with forced air cooling, you can't put the bare drive in a PC tower, it will overheat.

I cannot imagine archiving 20TB with my cheapo setup, but I also cannot imagine spending ~3K for a newer drive, this is a sweet spot for me. After buying a broken drive first, I am now always nervous that this drive will quit on me. It is definitely a passion project at this point.

Thanks for the info. For 1.5 TB I'd just use an extra drive.

My 20 TB is 11 x 3 TB used SAS drives in a raidz3 config. You can pickup a 3 TB drive off eBay for $18-$24.

I should've probably just bought four 10 TB WD red drives, but building this system has been, uh, a passion project I guess. :-)

It's still cheaper than if I'd bought new red drives, but I haven't factored in the electricity cost of running 11 SAS drives vs four red drives... the breakeven is probably less than a year. :-(

Sounds like you reinvented mbuffer.

http://www.maier-komor.de/mbuffer.html

Why not publish yours as well?

For other file systems (not ZFS), spd_readdir.so can offer a good performance boost.

https://kernel.googlesource.com/pub/scm/linux/kernel/git/djw...

It's probably possible to do the same on ZFS, but CoW certainly makes realizing this concept more difficult.

Tar not working for modern tapes by default is pretty ironic.

Not necessarily. Tar was invented way back, in early Unix days. I know it would have been updated sometimes, in GNU tar and other Unix versions, but they may not have anticipated the massive growth in tape capacities and speeds. Using dd with a high block size to write "virtual tapes" (as described elsewhere in this thread) from disk to tape might be one solution (not tried, just had the idea now).

It’s not exactly best practice, but instead of tarballing the files themselves maybe you would have better performance just writing a snapshot via zfs send direct to tape?

I admit to have thought of that, but using a zfs stream for archival would make eventual restore that much more complicated. A single bit error at the beginning of the stream would make the rest of the archive un-accessible, whereas with tar I can literally slice the tape and still recover data from it.

You can now tell ZFS to still process bad snapshots (with the risk of still losing some corrupted data).

long ago there was a utility called `buffer` that solved that problem. i know i've seen a "buffering netcat" go by here recently, that should serve too.

LTO speed has stopped doubling with LTO-4 - it's up to 400 MB/s uncompressed - which by the time LTO-15 is out it might be around 1GB/s or more.

Still a long time to write.

Tape libraries often have multiple drives and it's possible to write parallel to them.

Tbh, I think that in the near future we'll figure how to flip one bit worth of information in a single atom, perhaps with a high precision magnetic field, and use crystals to store amounts of data we don't have prefixes for yet.

Doubtful this is useful because it’s just not enough mass to be stable.

Not anywhere near room temperature, anyways. Plus it's not particularly helpful to store data in a single atom when it takes a much larger device to read the data. NAND flash cells are already about as small as they can be, but now they're being stacked on top of each other for better density. This is a much more promising avenue. Holographic storage also leverages the third dimension for huge increase in density compared to traditional optical drives.

What do you mean? I thought atoms in crystals are well localized and only oscillate around their average grid position.

I mean that noise in the environment will interfere with the state of your atoms too easily.

The state I was thinking about is stable electron configuration. If an atom has two such configurations, and can switch between them, that's one bit worth of storage.

And what if a stray photon should strike your atom? Does that not have a chance of changing the state? I don’t know much about electron configurations.

I suppose a random photon won't have the right energy to switch the state. But even when this happens, we have error recovery algorithms: cells in regular hard disks die all the time.

A single atomic electron state will almost certainly switch to a problematic degree except for very exceptional materials in typical room temperature type conditions - free ozone, UV light, background radiation. most crystals will grow or change at room temperature in some way over time (normalizing to the lower energy state if they weren’t 100% already, or high energy perturbances like UV light damaging the matrix and then ‘falling down’ the energy ladder again). Diamonds ablate through oxidation as well.

Error correcting helps of course, but we already have to do a LOT of error correcting to make even the ‘low density’ spinning rust drives we have now work. At some point you have so much noise, your error correction overhead is too high to make it worthwhile - which is the current issue with high q-bit quantum computers for instance

We’re a long way from having the materials science to be able to do this. Same issues plague quantum computing. Maybe some day though?

MRI machines do something like this. Probably not at the atom scale accuracy. UV light can be ignored because the crystal would sit in a opaque case. A trivial error correction is writing a bit to many places, the idea similar to bloom filters.

If it's sitting in a truly opaque case, it definitely isn't a tape! Shutters aren't perfect, and they open every time the cassette is used. That sounds more like a hard drive? UV and other short wavelength light can come from inside the case or the crystal from background radiation or cosmic rays as well. Reed-solomon encoding, n-copies, they all have capacity tradeoffs and error rate detection/correction limits, and get away from the 'single bit' ideal quite a bit (for n-copies, at least by 2x, usually 3x even in the most ideal situation). It's also non-trivial to avoid something like a background radiation emission 'tunneling' through and causing non-trivial damage to a sector.

It's going to be statistical level analysis for awhile, at least at temps near room temperature or devices only practical in a specially shielded lab.

MRIs are heavily shielded and regularly calibrated, and are only doing bulk statistical analysis. AFM or MFM [https://en.wikipedia.org/wiki/Atomic_force_microscopy] is the closest we can currently get, and they suffer from a number of issues, including damage to what it's touching, repeatability/precision issues, thermal drift, etc. and that is just getting what is usually a one time read.

We're getting closer to writing out the Akashic records. Interesting recursion of our present moments that it becomes observable.

Could that be mediated with many read heads at different locations?

Tapes only have one head. Otherwise, they couldn't wind or rewind.

It'd probably be mediated with a thicker tape with a thicker head: the thicker the tape, the more "data per inch" passes through the head, which allows bandwidth to increase. That would change the LTO-tape dimensions however, so that probably isn't an option.

Fortunately, by shrinking down the size of data (more data per square mm), the bandwidth necessarily increases. So maybe this technology will have a faster read/write speed than today's tech.

I don’t see a reason why you couldn’t have multiple heads. As it stands, in order to write a complete LTO tape, you make multiple passes across the tape. A second write head would allow you to make fewer passes across the tape to fill it.

The data on an LTO tape is written in a serpentine manner, or boustrophedon if you prefer, with overlapping series of tracks separated by the servo tracks used to position the head.

What would happen when one head can't write out data fast enough (for some reason) and the tape has to slow down and back up. Now, instead of one head, you have multiple heads that have to resync and get going again.

I dunno, if multiple heads were worth it, I think it would be common.

> What would happen when one head can't write out data fast enough (for some reason) and the tape has to slow down and back up.

The tape does not rewind in this scenario.

It is fairly common for the system to not be able to keep up with the full write speed of the tape. Just as a reminder, LTO7 has a write speed of 300 Mb/s, and filling a 300 Mb/s pipe for five and a half hours is no small task.

There are two ways the system deals with it. First, the drive will slow down the tape to match the data rate. Second, the drive can mark a section of tape as failed and then rewrite that data to the next part of the tape. Instead of rewinding and writing over a section of tape, you’re using up more tape this way.

Tapes, like hard drives and SSDs, come with a certain amount of extra capacity to allow for this as well as other write errors.

> I dunno, if multiple heads were worth it, I think it would be common.

I’m sure that it’s “not worth it”, yes, absolutely. But I don’t see major technical hurdles here.

One technical hurdle would be figuring out how to, say, fill an 800 Mb/s pipe for six hours straight. That’s hard. It gets harder when you realize that a single tape library might have as many as 64 or 80 tape drives. 64 drives, multiplied by 400 Mb/s, is 26 Gb/s.

If you have a 64 drive tape robot, you probably have 65 computers: one controlling the load/unload mechanism, servicing 64 servers each dealing with one drive. (Or perhaps 16 servers that each handle 4, or 8x8.) Each of them has its own high speed drive for cache and independent NIC.

That's part of the solution, yes, but it's difficult to get the whole system set up this way and achieve something near maximum throughput. My understanding is that most people who use tape don't get anywhere near maximum throughput over the long term. Naive use of drives is still quite common... things like backing up with tar, or dumping a database straight to tape, and it's easy to imagine buffer starvation scenarios.

Returning to the idea of increasing throughput on the drives by adding heads, I'm just doubtful that it would be the best way to improve overall system performance. For alternatives, you can either get engineers to optimize usage of existing drives or buy more drives.

Reply on news.ycombinator.com