Comment by klodolph | Hacker News Reader

Comment by klodolph | Hacker News Reader

Parent Full thread

> Where did you get the gear?

I never ended up using it personally, but when I was considering it, used gear on eBay was top of the list.

> How cost-effective is the overall setup compared to spinning rust?

For personal use, it is not even remotely cost-effective unless you are a data hoarder. The cost of the drive is simply too high. Also keep in mind that the drive can become damaged or need servicing.

It only makes sense if you are dealing with e.g. large quantities of video. The article cites a minimum 140TB where tape becomes cheaper but this involves some dubious decision-making.

> How is driver support in popular operating systems?

I have only ever used tape with Linux + custom software so I can’t speak of software support in general.

> Is there more you'd recommend taking into consideration or learning before jumping in? Especially beyond the obvious realities of extremely high latency, of course ;)

The obvious reality to me here is that it just doesn’t make sense unless you’re in spitting distance of a petabyte. The only thing I can imagine someone doing personally is video, and it would have to be a lot of video like some kind of long-running documentary project that you are producing in your spare time.

The other factor is to be more paranoid about mixing tape generations than the vendors suggest. Ignore the specs and stick to same-generation media and drives.

Efficiently writing to a tape means filling e.g. a 20 MB/s write channel for LTO-6, or 750 MB/s for LTO-8. Can you sustain 750 MB/s? It’s not so easy in practice. You throw away tape capacity if you get buffer underruns.

> Can you sustain 750 MB/s? It’s not so easy in practice. You throw away tape capacity if you get buffer underruns.

The drive doesn't pause while the buffer fills again? Considering the price of the drives, I find it very surprising that they don't do that.

No, they stop, rewind, and accelerate again, causing massive mechanical wear.

The higher-end tape devices are capable of running at slower speeds to better match the rate you're feeding data to it to avoid this.

Yes. But it was argued that they could just stop and wait for the buffer. As said, they do, but you don't want them doing so.

> The drive doesn't pause while the buffer fills again?

The drive does stop, and it runs at different speeds depending on how fast you are feeding it data. However, this process wastes tape capacity as you switch speeds.

Thus you should pick one speed that's fast enough and stick to it.

I don't understand how stopping or switching speeds wastes tape capacity. Are you not talking about storage capacity but some other type of capacity?

> Can you sustain 750 MB/s?

Do 10% of a tape at a time, buffered on a $200-$300 NVMe SSD?

That wastes tape capacity when the drive inevitably has to stop and restart.

It's cheaper to run the tape slower (you can adjust it so that it is very close to what you can sustainably deliver)

> That wastes tape capacity when the drive inevitably has to stop and restart.

Not meaningfully, unless there's something ridiculous about the format I'm unaware of. A full tape write covers 160-200 linear kilometers. Stopping 10 times, which is once per hour, is barely anything. I haven't found anything listing the exact size of gaps, but I did find a post from someone having thousands of buffer underruns that was only losing half a second of capacity per underrun.

As Dylan16807 also said, you waste very little. I am using 6 TB LTO-7 tapes and I group the data written to the tapes in files of approximately 50 GB, which are read from a NVMe SSD.

This means that each 50 GB file is written at 3 Gb/s, the maximum tape speed, but after each 50 GB the tape is stopped then restarted. So I write on each tape about 120 files with an aggregate size of up to 45 905 860 blocks of 131 072 bytes each. That is more than the 6.0 TB advertised for the tapes, so I do not waste any measurable tape capacity. The 50 GB file size is chosen because it fits inside the 64 GB RAM of the computer with the tape drive and certain algorithms that I am using to process the tape files (lrzip & par2, for compression & redundancy) are faster when the entire file fits in the RAM.

Reply on news.ycombinator.com