Hacker News Reader

Comment by londons_explore | Hacker News Reader

Are there any tapes that offer "read and write simultaneously" functionality?

A common usecase for tapes seems to be to store a big archive of data. Imagine having some API to my archive, which says for each record/file stored whether I want to keep it stored, read it, or discard it.

Over time, many tapes start to be only half full of data I want stored (the rest has been discarded). It makes sense to read all the important stored data, consolidate it, and rewrite it to a new tape, perhaps with some new data.

For this usecase, you only need half as many tape drives if a drive has the ability to read data while rewriting a new data stream.

View on HN

I don't know about the drives themselves, but a company I worked for once used large, robotic arrays with 16 drives each. Typically they ran 15 drives writing (backups) and left one idle for reads (restores) as needed. The tapes were on the order of a kilometer long, so considering the time for a robot to pick the tape and seek to the proper spot, latency is pretty high in the best of circumstances (no one else is using the one reader drive when you request it).

But the drives that are doing writing... Either they are writing onto an entirely new tape, or they're writing over an existing tape. If it's an existing tape, you need to make sure that everything on the tape is not needed anymore. Yet commonly 75% of the data on the tape isn't needed anymore (retention period expired), but some might still be needed. Unless you have a hard drive somewhere with another copy of the data, you need to read every tape before rewriting it.

I suspect your use case is strictly for backups, whereas my use case is more archival (where all copies of the data are on tape, and there are no copies on hard drives).

They can also append data to an existing tape.

These tape libraries can also defragment tapes. For this all the data that needs to preserved is read from the tape and written to another tape.

Even for backups you have a consolidation/defragmentation task running every so often that frees up space on the tapes and only keeps the backups you specified in your retention policy.

While that isn't impossible, nobody will do that. Disks are divided into sectors, with "a lot" of empty space between each sector - random variance makes different sectors take up different amounts of space on the disk. (minimizing this is important to disks, but the nature of physical media is you can never fully eliminate it). With a tape you don't have sectors, and so no blank space. As such as soon as you decide to delete one record the entire rest of the tape needs to be re-written.

There have been people who put a filesystem on a tape with sectors - they could then do this, but since tapes are always sequential access it isn't practical, just an interesting hack to show off.

Log structured file systems solve this problem[1]. They can be extended to tape libraries in various ways. Tapes are still rewritten, but not nearly as often as once per deletion.

[1]: https://en.wikipedia.org/wiki/Log-structured_file_system

After few years, like 5 years to 10 years, you will want to rewrite the data to a new tape. Even LTO tapes will age and go bad after a while. You don't want to be the one who finds out the hard way how long LTO tape actually lasts.

And there is no avoiding reading all the data on the tape when rewriting to another tape. The media is cheap after all. And therefore "discard" is largely pointless, unless it's 90% or more of the data.

I guess you'd often need to defrag the tape?

Reply on news.ycombinator.com