Cassette tapes are the future of big data storage
newscientist.com
newscientist.com
This quote:
The downside of tapes is that they are slower to access than hard discs because they have to be fetched by a robotic mechanism, inserted in a reader and spooled to the right point. But the Linear Tape File System, which is being developed, expedites this process to make it comparable to disc drives, Eleftheriou says.
indicates that the reporter did not understand what he was being told. LTFS offers an abstraction layer so that software can treat a tape drive like a hard drive. That makes programming easier, but it will not have any effect on performance. A file system cannot, after all, magically bestow random access unto a sequential access medium.
So if tape is to make a comeback in storing data for, say, the web (as implied by the article's opening paragraph, then it's still going to have a HDD based system in front of it as a cache. It would be an interesting future where viewing a post that hadn't been accessed in a few years took several minutes as you waited for that old data to be recached from tape.
> It's fascinating that tape drives are making a comeback, but to suggest that they can replace HDDs is just a fantasy
Tape + RAID vs. HDD with no optimizations is a useless comparison. Why would anyone even care about that?
With that said, I outlined in another comment[1] why a mirrored array of tapes would have pretty huge drawbacks regardless. Even if you have an expensive array of tape drives (as opposed to one drive with many tapes), a mirrored array will drag your write seek performance toward a constant worst case as the array grows. The same is true of RAID 1 for HDDs, of course, but the worst case seek time of an HDD is orders of magnitude less than that of a tape drive, so HDDs still win.
In any case, if you look at the details of the situation, the answer you come to is pretty boring: tape drives can compete with HDDs in a small slice of real world cases (often when used in conjunction with HDDs), and higher density tape drives will slightly widen that slice.
If you have a situation where you are writing a ton of data, but almost never reading or erasing, then high density tapes will play well to that situation (sans mirroring).
If you have a situation where you are writing a relatively small amount of data over a long period of time, reading it back comparatively frequently, and never erasing anything so that your total storage needs become very large, then maybe a mirrored array of tape drives would make sense, if it were behind a sizable HDD array. But it's a stretch.
When the data set is so large that maintaining it all online at once is impractical and the nature of the data makes continuous and immediate access unnecessary, then tape may solve the problem.
I'm not saying that future tape technologies will be as fast as the hard drives of the equivalent future - linear access has inherent limitations. However, there are relatively simple ways in which data access could be improved and such improvements (and some tuning) could make tape adequate for some applications.
You generally have two bottlenecks in tape seek performance. The first is the fetch from the robot, which can take seconds. The other is spindle rate on the tape drive itself which is limited by the durability of the tape. Also, in order to make this work, you'd need many tape drives, with many robots accessing the same racks of tapes. There's a physical limit to the number of drives and robots that can generally fit in a cage and have access to the same tapes so you'll have a lot of trouble pulling this off.
All this and you still haven't addressed the difficulties involved in overwriting tape. Nor does it address the normal redundancy you already need to deal with corrupted and broken tapes, or the fact that you'll need to expand your setup even further if you want multiple users of the filesystem at once.
Current systems already do caching and concurrent reads off multiple tapes. Tape is still not a drop in replacement for disk. Any application you write for it has to have it's limitations in mind.
Considering the seek time of the robot arm. Striping data across four tapes just means you need to wait for the robot to grab four tapes.
Although brudgers talked about writing at different offsets, that's actually pointless if you don't rewind each tape between accesses (and why would you?). Assuming random access patterns, your read speed on the array will, on average, increase linearly with the number of tapes. This will come at the cost of write latency, the magnitude of the cost depending on how you implement the array.
If the array has independent drives (which would be very expensive, but let's assume for the moment), then your write latency is always dependent on the longest seek (you have to wait for the write to occur on every tape). That means that as the number of tapes increases, your average writing latency will approach the worst case for a single tape. In other words, with an infinite number of full drives, you would always have to wait for one of the drives to seek across an entire tape. That's pretty bad news, but it's acceptable for some cases.
On the other hand, if you had one drive and a tape library, your write times would really be completely unacceptable. Your linear read seek speed increase would come at the cost of a linear write seek time increase, plus a linear increase in writing throughput time.
Ugh. You're more right than you know. This is a misconception that I sometimes have to deal with at my work. Tape systems are not harddrive systems and can't be treated as such even if you use LTFS or an HSM. The problem is that even though you can delete things by removing them from the (disk based) inode table, you can't actually remove them from tape or overwrite them without a special effort, which means that you can't simply reclaim tape space.
Being able to access a tapes with the standard API used for HDDs is convenient but you can't lose awareness of the underlying system or you'll screw things up mightily.
Also, stay away from DiskXttender. I've used that product and it's terrible.
Where does the gain in storage density come from, New Scientist? This article is so breathless to capture the linkbait of using old technology that just about any useful info has been left out. The prototype, at about 4" x 4" x 0.75", is just larger than a desktop HDD, and can hold about 8.5 times as much data as a 4TB 3.5" HDD.
Does barium ferrite allow for tighter magnetic fields, which would allow for a higher data density? If so, is that material being used in spinning disk HDDs? And how does this extremely physically complex system stack up against SSDs, as they lower in cost?
SSDs would be the opposite of tape. Big$/GB low latency vs. Low$/GB high latency. Obviously SSDs would replace almost all other media if it could beat tape on $/GB.
And here is a scientific paper on Scaling tape- recording areal densities to 100 Gb/in2 http://signallake.com/innovation/argumedo.pdf
huh?
ADDED. Withdrawn: see child comment.
Wouldn't it be more efficient to analyze it, save the interesting data, and get rid of the rest?
The data storage costs are a pittance compared to the cost of telescope time. By saving the data you're getting the most out of each observation.
the other approach - taken with something like the SDSS - is to make a telescope for just one task (typically a survey). then you can process the data and throw it away. i am pretty sure that is what SDSS did, and what, say, the LSST will probably do too (if it ever gets finished).
the advantage of the first approach is that you are much more flexible, which means more likelihood of making a big discovery (particularly when, as with this telescope, collecting area is larger than ever before, which means that you can see fainter and further back in time, making it ideal to study isolated, unusual objects - in contrast, survey telescopes make different technical compromises so that they cover a wider field of view), and also more chances to make and exploit upgrades over time. the downside is dealing with issues like this (another issue is data transport - traditionally you go to a telescope and then take the data home with you; i imagine we're now getting to the point where instead you will process data local to the data storage).
disclaimer: i was an optical astronomer, not radio, and that was years ago, so this may already be old news / incorrect in details. but the general idea should be ok.
http://en.wikipedia.org/wiki/Square_Kilometre_Array http://www.sdss.org/ http://www.lsst.org/lsst/
ps often telescopes do make data public after a certain time. but the idea is not so much to allow reanalysis as to make sure the people who originally took the data reduce and publish it. it's easy to postpone that kind of work, but the idea that someone else might do it and publish first is quite a motivator.
There's also a long history of data being mined later for things which the original experimenters didn't even think to collect - e.g. http://www.nasa.gov/mission_pages/hubble/science/elusive-pla... - or types of analysis which was considered too expensive to perform in the past (e.g. biology went from a pre- to post- informatics era when the unit of work went from “single grad student” to “many-thousand node cluster”). Given how expensive the data was to collect, it almost always makes sense to see if it will lead to any other value.
Thanks for the insight, New Scientist.
The most interesting item in that story was the claim (probably incorrect, based on the article's overall sloppiness) that LTFS will be "comparable" in speed to hard drive access.
Seriously, though, I think that this is great news. If this really gains any traction this could represent a resurgence for some Japanese companies which hold patents on cassette tape technology. After the recent Fukushima debacle, they could certainly use it.