Sony and Panasonic announce the Archival Disc format | Hacker News Reader

Sony and Panasonic announce the Archival Disc format

sony.net

152 pointsjhack147 comments

Important information that will drastically affect actual usage not provided: - expected and guaranteed lifetime of discs - minimum undamaged read/write speeds - recoverable-error read speeds - bit error rate for writing and reading

Let's take a typical small business system's requirements. We have a 2U database with 12 3.5" hot-swap disks, 2 200GB SSDs for caching and 10 3TB disks in a RAID10. We have up to 15TB that we want to archive.

Optimism: we get 100MB/s write speed and a write BER of 10^-15. We swap 30 500GB disks, each taking about an hour and a half to write, and about one in a thousand disks has an unrecoverable error.

More likely: we get 75MB/s sustained write speed and a write BER of 10^-14. We swap 30 500GB disks, each taking close to two hours, and we can expect one in three complete sets to have an unrecoverable error.

Press releases: not quite useful.

In the meanwhile, you could grab an existing LTO-6 drive, and archive everything on 6 uncompressed tapes in about 30 hours (much less if your data is prone to compression).

After building a very very fast staging disk system to feed it. That's one of many reasons I went with LTO-4 and a slower HP tape drive, I can feed it without shoe-shining from one 15K SAS disk that I have Bacula staging to ... and I don't care about how long it takes to do my backups or restores, which is generally not the case for businesses. Back when I made that choice, it was said no single magnetic disk could keep up with an LTO-5 tape drive.

Or 4x 4TB drives (at ~$180-$250 each from a quick newegg search), at ~15 hours if you can connect all of them at the same time, or ~60 hours if you need to do that serially (assuming 75MB/sec, which I often get - are you sure you're actually getting 150MB/sec on your LTO-6 tapes?)

At about a quarter of the price of the LTO tape drive alone -- data can be recovered anywhere for the next 20 years or so (I have tapes from 2002 that I was unable to find a tape drive for in 2007, but I have a pair of 15GB IDE disks from 2000 that I checked last month and are still readable).

At $60-$80/2.5TB, ($24/TB) LTO6 does have an advantage to disk ($40/TB) if you have hundreds of TBs to backup. However, because of the cost of the drive itself ($3000), you only break even after 200TB or so. And if your drive breaks (or you need to backup in one place and recover in another, as is often the case with DR plans), you break even only after 400TB or so.

Since approximately 1998, hard drives for backup beat tapes on capacity, price, access speed (drives are random access!) accessibility (an external universal SATA/IDE can be bought for $30 or so, but you already have the connections on your motherboard so you don't even need it). I haven't been able to find anyone with a convincing argument for tapes - maybe you have one?

I'd be more interested if you made your case with LTO-5 tape; the latest and greatest generally comes at a price premium.

I switched from USB 2 external hard drives to LTO-4 (see my other comment in this thread for one reason for that generation, which BTW per the LTO spec can be read by an LTO-6 drive) because I wanted multiple copies of my data in a less physically fragile form than hard drives (something I'm willing to routinely schlep to a safe deposit box). And I'm an old timer, bought my first DECtape in 1978 (sic), because it was neat, and then it became handy when I learned the -rf flags to the rm command ^_^. I'm very used to tapes, how they work and don't, what you can and can't do with them. And while I was flirting with hard disk backup almost had a catastrophic loss (http://www.ancell-ent.com/1715_Rex_Ave_127B_Joplin/images/) only redeemed by luck and offsite storage to rsync.net in Denver.

I didn't buy a high end, super fast tape drive, rather an internal HP 1760 EH919 drive, Newegg will sell you one at 2,130, but with some bargain hunting you can get for $1,400 easily (we managed $1,200), and Fujifilm and HP tapes (wouldn't trust any others, except maybe IBM but those are expensive) from them are going for $39 to $42 quantity one, you can of course get better pricing if you buy packs or 20 (or more) and/or go bargain hunting, e.g. 35.45 for the first Amazon Merchant I found with a rating >= 96% for a 20 pack of HP tapes. That's for 800GB uncompressed storage per tape.

Per my other comment I also had to buy a 15K SAS drive, that was $250 back then (late summer of 2011).

And I should note one of the biggest use cases for LTO drives is fast backups for businesses. With of course plenty of $$$ backing hardware to keep LTO-5 or LTO-6 drives happy.

Case is not much different for LTO 5: drive is a little cheaper, as you note. Media is slower and almost same price per TB.

You still need drive redundancy in most setups. And my experience with tapes has been that they are not significantly more robust than drives: they break and die when they fall on a hard floor, they get ruined if heated too much.

And an issue that might be gone in the recent LTO era (has it?), but which I met several times: if you need to recover tapes made 5 years ago, finding a usable tape drive for use or sale might be a big problem. The last drive I had I can't connect to a modern motherboard was an MFM controller-RLL drive pair I had in 1987. IDE came shortly after.

See wazoox's comment: per the spec/in theory and in practice for him, you can read back 2 generations of LTO tape (and write back one, to ease transitions).

Any sane archiving plan will read old tapes "before it's too late" (yes, I know, I know) and re-archive them. I suspect a prime use case for this proposed optical media is archiving what Hollywood et. al. does, which is getting to be rather large (10s or 100 TB per a major film?), put them in controlled environment facilities and then forget about them unless the film is such a success you want to make a new release that needs to go back to the masters, or more material for a "Director's Cut" etc.

> Any sane archiving plan will read old tapes "before it's too late" (yes, I know, I know) and re-archive them.

That's true. In fact, any sane archiving/backup plan REQUIRES that you regularly verify the integrity of the data, preferably by restoring -- and refreshing that data onto new media "before it's too late". That is true for both tapes and drives.

A previous employer had significant failure rates until they realized that the LTO tapes were being flipped sideways at some point during storage: simply failing to keep them flat pushed total tape failures well into double-digit rates over a year!

Look here: http://www.clir.org/pubs/reports/pub54/estimation_of_LEs.htm...

Lifetime of tapes may not be what you think it is.

10-20 years in my environment (60-95% humidity if not controlled) is what I expected. And that's what I've been getting from older left-in-storage drives since 2000 (all of them still ok).

While I'd hope professional backup tape is a little better than high end early 1990s VHS tapes, the 10-20 years that beagle3 expects is also in line with what is generally expected out of modern backup tape. Granted, we are talking about archival storage, and it is still used for that, but after disks got big enough that tapes were seldom used for more than backups the pattern is to continually refresh them with multiple backups from disk.

It may be better, or worse. But temp matters a lot - several data centers I've helped in are often left to get to almost 80 degrees - I can see that being a huge problem on their lifespan!

I know from experience that people are very likely to think "Tape?! It'll last forever!...right?" which is dangerous.

Well I've recently restored 150 LTO-2 tapes from 2002/2003 using a perfectly current LTO-4 drive, and I'm pretty sure you'd have a heck of a hard time successfully reading all of a bunch of 150 hard drives of the same vintage without losing a single file.

Hard drives are much less reliable than tapes; hard drives aren't made to be stored on shelves; and when a hard drive fails, you most of the time lose most or all of its data, while a tape failure generally affects only one file (if it's a reasonably large file and you aren't using compression).

As soon as your storage and archival needs are more than a few disk drives, do yourself a service and switch either to RDX or to tape. Storing basic HDD on shelves is a recipe for data loss. I know it because we sell backup solutions to people doing it and losing data all the time :)

>> Hard drives are much less reliable than tapes; hard drives aren't made to be stored on shelves;

I can back this up. For some reason, someone decided to back things up to hard drives and even though they are stored in anti-static bags in a 600lb fire safe, when I remove the drives from 1 year ago to re-write with that months backup, I've had up to 1/2 the drives not spin back up. Ugh.

Disclaimer: my experience with tapes is dated. I used IBM tapes with a mainframe (System 390, library robot and all) until 1995, and then only used consumer level myself, but had witnessed the occasional horror story at a client's.

> Well I've recently restored 150 LTO-2 tapes from 2002/2003 using a perfectly current LTO-4 drive, and I'm pretty sure you'd have a heck of a hard time successfully reading all of a bunch of 150 hard drives of the same vintage without losing a single file.

That might be true. However, I have so far had perfect success with about 50 or so drives that I've restored from (across 10 years or so), and abysmal success with tapes (about 1/10 in the mainframe days would not restore).

> Hard drives are much less reliable than tapes; hard drives aren't made to be stored on shelves;

That's true, but neither are tapes. Hard drives are less tolerant to environmental conditions than tapes, they survive better when exposed to heat, and get damaged more quickly when exposed to cold (oils used to keep it running smooth tend to congeal irreversibly when exposed to cold for long). Also, at least in the past, even weak magnetic fields wreaked havoc on tapes, and even strong ones spared drives.

> As soon as your storage and archival needs are more than a few disk drives, do yourself a service and switch either to RDX or to tape. Storing basic HDD on shelves is a recipe for data loss. I know it because we sell backup solutions to people doing it and losing data all the time :)

In my current top-secret venture, I need to keep everything (~20TB/customer) randomly accessible for 3 years, and cold-store accessible or 7. So far, we just keep multiple copies (some online and some offline), and keep refershing to newer larger drives every couple of years. When I specced it 3 years ago, tapes were twice as expensive (I need a tape drive at every customer location + at least two at my office) and didn't properly address the random access aspect.

But I keep re-evaluating. So far, the next step seems to be going to a BackBlaze style pod - which would also address the valid drive-fails-completely concern you raised.

p.s., I find the "compressed" statistics really misleading. Every kind of backup software can compress - if I saw a hard drive manufacturer sell a ("2TB drive (1TB uncompressed)" I would consider it fraudulent, but that's the standard for tapes. For the record, my data is (as far as tape drives are concerned) not compressible at all -- think audio, video, jpegs, and other analog recordings.

EDIT: p.p.s: Of course, don't keep hard drives on shelves. Nor tapes. Humidity and temperature need to be controlled, and the environment needs to be anti-static. For archival purposes, the 5400-RPM usb self-powered disks are a little slow, but keep very well, in my experience, with reasonable temp/humidity.

EDIT2: (bad) memories are coming back. The biggest problem I recall with tapes in the mainframe days, were that some tapes (the daily and weekly backup sets) were continuously reused, to the point that mechanical wear and tear was a bigger problem than magnetic wear. And the even bigger problem was that you only found that out when trying to restore. One advantage of disks is that you keep reading them while writing them - wear is apparent much more quickly. If you're only doing archives and never rewrite tapes, that doesn't matter. If you keep reusing the same tapes/disks, it makes a huge difference in reliability.

Yes modern use case for tapes is archive, not backup. In fact, LTO-7 should be back-compatible from LTO-4 onwards to reflect this. Most people do backup disk-to-disk, with a remote copy on tape for instance.

> do yourself a service and switch either to RDX or to tape. Storing basic HDD on shelves is a recipe for data loss.

What makes RDX (basically, an HDD encapsulated in a rubber cartridge) more reliable than e.g. an off-the-shelf rubber-USB-enclosed HDD? It's not like cartridge makers are making hard drives themselves - they source them from the same manufacturers as everyone else.

All the complaints about HDDs for backup/archive apply to RDX, as far as I can tell. And at $100-$150/TB (and I can't even find cartridges bigger than 1.5TB for sale anywhere), it seems to combine with the worst price as well.

At least the RDX connector is made to withstand several thousand insertions (USB clearly can't). I suppose that the makers take some precaution to give some shelf durability to the drives within, too, but that's just a guess.

I am unconvinced.

If your archival media is going through thousands of insertions, then it's not archival - it's the most inefficient swap device ever.

But I've connected/disconnected my laptop's USB mouse and keyboard a couple of thousands of time (several times a day over the last few years) and neither connector nor socket seem to be suffering.

Furthermore, most of my cold store drives are of the self-powered USB 5400 RPM variety (getting a decent 30-40MB/sec on USB3) - if something breaks, I replace the $1 cable that connects drive to computer.

Seriously, RDX seems like an awful choice - does someone here actually know differently?

Yeah, it's not at all clear to me that USB connectors aren't worth "thousands" of insertion/removal cycles. They look very well thought out, plenty of contact area. Looks like it depends on the thickness of the gold contacts, per this supplier: http://www.globalconnectortechnology.com/usb-connectors/

  USB2.0 Full Size: 1,500 mating cycles, gold flash plating.
  USB2.0 Mini: 5,000 mating cycles, 15µ” gold plating. 
  USB2.0 Micro: 10,000 mating cycles, 30µ” gold plating.
  USB3.0 Full Size: 5,000 mating cycles, 30µ” gold plating.
  USB 3.0 Micro: 10,000 mating cycles, 30µ” gold plating.

With an added comment:

Why Micro types offer better durability?

Accomplished by moving leaf-spring from the PCB receptacle to plug, the most-stressed part is now on the cable side of the connection.

Inexpensive cable bears most wear instead of the µUSB device.

Plus as you note, it's easy to make it a cheap replaceable part. Use a USB extension cable on the computer side and you won't be wearing that out (I've done that trick in times past).

What about M-Disc? The DVD-writers that can write them are just as cheap as normal ones and I think $4 for a DVD-R is acceptable for long term storage. They announced that they would offer Blu-Ray media soon but I haven't seen them for sale.

http://www.mdisc.com/what-is-mdisc/ for those curious - it appears to be a stronger archival DVD/blu-ray that is already burn-able on certain drives that are on the market.

Has anybody used these? A 1,000-year DVD/Blu-Ray sounds good. I would use it if it is actually reliable. (Actually just want decades, but centuries is better.)

Non LTH BD-R has an estimated lifetime of 50 years if stored properly. 40 cents a disk. In a cd case, in your closet.

M-Disk 1000 years, same storage idea. $5 a disc.

This makes perfect sense.

Sony and Panasonic need to do something with their lead in optical disk technology, but there's no demand for a Blu-ray successor. Heck, there isn't even much demand for Blu-ray. So archiving is the way to go.

I just hope they make those things to last.

Only one problem -- archiving is mostly a big business use case. Therefore, the technology will only have big business price tags. Remember when you could back up a home hard drive onto a single tape, and the media cost was only a fraction of your hard drive cost? And the tape drive was no more expensive than the hard drive? (i.e., those mini QIC cartridges).

> Only one problem -- archiving is mostly a big business use case.

Not necessarily. With photography having been digital for about a decade and home video for about half that, tech savvy families easily have several TBs of data that they would want to hold on to for the long haul. The prospect of being able to burn a complete copy of these things on a few discs and leave them at the grandparents or in a bank box seems very appealing, and only more so over the coming years.

It's easier to imagine those families moving their data to ever larger magnetic disks than making something like this "consumer grade" (as in "consumer-proof").

Most people I know just copy everything to a large external USB drive (sometimes managed by backup software). So these things would have to be less than $50/TB for the media, and no more than a couple hundred for the drive, in order to compete with external USB spinning disks.

After they knock the external drive off the desk, they might go looking for a different media for offline storage. The cloud works for some cases, but it's definitely not great for large data volume- at least in the US, with broadband that isn't nearly what it could be.

The nice thing about external USB drives is that you can position them some distance from your computer. Back before I switched to LTO tape, mine were taped down in a low/no traffic area. But, yes, they're fragile, almost certainly the most fragile backup media you can use, and my first set didn't survive this http://www.ancell-ent.com/1715_Rex_Ave_127B_Joplin/images/, at least not without reporting so many errors I gave up on them.

And you're of course right about the cloud, especially in the US; I've got rock solid AT&T ADSL, but it's very A(symmetric) and has a 150 GiB monthly cap, so only my very most important stuff gets saved to rsync.net, which e.g. saved my email from the tornado.

Hard Disks are not an archival format. In addition to the standard data retention issues... arms can rust and get stuck. There are simply more moving parts to a mechanical hard drive than either optical storage or tape storage.

Archival hard drives exist of course, but are at a cost disadvantage compared to tapes or optical media.

At least while we're waiting for bigger SSDs. Not sure what the expected lifetime of a (theoretical) 4TB SSD that's used mostly in append-only manner is. Will the drive outlast the USB3/SATA interface it comes with? It should at least be pretty resistant against physical wear-and-tear.

The storage cells are essentially capacitors, they store a bit of electrical charge. Eventually that'll leak away, unless the controller is sophisticated enough to do DRAM style refreshing every N years. Or you do it.

SSDs are way too young to depend upon, as far as I'm concerned. For that matter, if you really screw up and have to recover data from a drive, they're much more opaque than hard disks.

Archiving is very much a consumer use case. Photos/videos, and generally old computer HD contents are the main applications.

So little demand for Blu-ray is there that Panasonic's own archival storage device reached End Of Life four years after its launch in 2007. That one was called PowerFile [1].

The idea was pretty good, the industrial design was lackluster but serviceable. But the longevity of the discs was questionable (especially if you took advantage of the ability to remove the spindles holding them), and the devices themselves were abruptly pulled from the market around 2011.

The drives inside were actually laptop drives--the kind you used to slide into your laptop before you had a MacBook. I doubt they would last very long, but we didn't really get to find out, because once new units were no longer being sold we had to abandon the system.

So, do not buy this thing for long-term storage, because one of its main backers already proved they do not have what it takes to service this market.

[1] http://www.kintronics.com/powerfile/prodSubIntro.html

Very good point. I in general no longer trust Sony as a corporation, although I haven't heard any horror stories about their backup products. That Matsushita, ah, I'd forgotten they renamed themselves Panasonic in 2008, would pull such a stunt ... nope, not going to trust this stuff for a decade. Will stick to LTO tape and Taiyo Yuden CD-Rs.

Demand for Blu-ray is there. Perhaps not as great or immediate as the demand for DVD after VHS was (thanks to great backwards compatibility), but at least everyone I know or have met has switched to Blu-ray for their new movie buys.

If there's any repression in the movie market, I would blame prices/multi-format bundling. I don't want to buy a six-version pack with two different digital versions, a Blu-ray copy, a DVD copy, a 4:3 DVD copy, and a VHS copy for $40. I want ONE DISC, Blu-ray, for $8-10

Name a bundle that included BR, DVD and... Widescreen/4:3 copies? Much less VHS?

I'll pay a couple extra dollars for the BR + DVD bundles (Although, all my stuff plays BR these days)... I think it's fairly standard to get Widescreen these days as well. I don't remember any recent 4:3 releases for newer titles.

The main issue I have is "release day" discounts of $~10-15... followed by months of $~20-25. I hate missing a "must have" and then waiting until the price is reasonable again.

I was mildly exaggerating, but I mean stuff like this:

http://www.amazon.com/dp/B00CTSDDVO/ref=wl_it_dp_o_pC_nS_ttl...

http://www.amazon.com/dp/B00867GHS8/ref=wl_it_dp_o_pC_nS_ttl...

I don't mind the 2-disc BR/DVD bundles, I just wish I had the option to buy a 1-disc BR instead. It gets even harder when you want a 1-disc 3D BR.

What these bundles are telling us---and they're fairly common in US market anime---is that the price of pressing a DVD is so cheap it's better to make one SKU with both. Probably also encourages people to buy a Blu-Ray player as well if they still haven't.

And that's fine, but the bundle seems to be used to justify higher prices. "Oh, well it would be $10, but you get both a Blu-ray and a DVD, so we have to charge you $20"

Bleah. I haven't noticed that in the US anime market, but prices are all over the map as companies seek better/sustainable business models. Anime suffered from the general decline in DVD sales, and the decrease in retail outlets, like The Musicland Group (Musicland, Sam Goody, Suncoast Motion Picture Company, On Cue, and Media Play superstores) going poof after Best Buy bought them.

> I hate missing a "must have" and then waiting until the price is reasonable again.

This tells you the item was not a "must have" then, or you would have bought it already, rather than waiting...

Well to be honest, no Movie is a "Must Have"... for me at least. But a "I REALLY want that to sit on my shelf and hold up all the dust" is only worth so much.

you're cherry picking your sample set, here. of course it makes more sense that people who still buy movies will have a significant percentage who adopted blu-ray.

personally, i know not one single person who still buys physical video discs anymore. personally, i can say that demand for blu-ray is non-existant.

what would be useful, here, would be some sort of data that compares usage/adoption rates for units of time after a formats release. how many people are using blu-ray X years after it became available, compared to DVD, etc.

i would expect it to not be as popular as DVD or VHS before it, but expectations and anecdata aren't terribly useful.

I expect this will be the same form-factor as existing 5.25" magneto-optical and Ultra Density Optical disks, which have been around for quite a while.

Back in 1998, I dealt with HP jukeboxes the size of a wardrobe with robotic arms to pluck 9.1GB magneto-optical disks out of the racks and stick them into drives. IIRC, HP guaranteed their disks for 100 years.

I get a kick out of the idea that someone would guarantee something beyond their lifespan, and possibly that of the institution. 100 years is still a pretty long time, and lots of things can happen, so I'm not sure who would be there at the 80 year mark if something were to go wrong with the disks.

"Recall this batch immediately! These disks have a defect that will make them last 80, 90 years tops!"

Even if someone is around, what exactly are they going to do about it? It's not like they can wave a magic wand and get your data back.

I guarantee my stone tablets will last 100 years if they are cared for as specified in the user manual.

Am I crazy?

Yes stone is great, but how many bytes/tablet do you get on average?

Are you chiseling in UTF-8 or UTF-16?

I've read that MO is much more stable/reliable than writable DVD/BD. I spent a lot of time tracking down certain Taiyo Yuden discs to try to maximize the chance that I'd be able to read the data from DVD-Rs a couple of years later. I've heard a lot of negative things about BD-R reliability. Did you find that MO was better in terms of lifetime?

Taiyo Yuden DVD recordable media as of a few years ago cost 1 cent more than their renown CD-R media. I don't trust it for anything serious, I barely trust MAM-A Gold DVD-RWs.

I played the MO game in the '90s and found them to be rock solid, and durable as well (e.g. I'd throw a disk cartridge in my backpack without problems). Theoretically they looked quite quite solid.

From the 2013 release: "they intend to offer solutions that preserve valuable data for future generations".

No mention in the new release how long they are meant to last though. This seems like a pretty key point to me.

Even if they are not meant to last, say, more than 10 years, you could back them up with fresh disks in 8-9 years timeframe and be good to go for another decade, What's important after is that the disk can still be read by newer drives later on.

This does not appear to be a consumer-oriented format like CDs and DVDs, but rather a niche product. Meaning that it will probably be not cheap. How is it supposed to be competitive with tape? 1TB is not that much.

Tape is less reliable than optical disks.

It's also so slow to retrieve the content that as a DR strategy it's not very good, since you could be waiting months to recover all your data.

I would imagine that a silo with multiple robots to place disks in many readers would be significantly better than tape in most ways, and would retain the low power advantage that tape enjoys over powered systems.

The time to read the complete tape might be longer than the time to read a blu-ray, but in terms of data rates modern tapes are still about a factor of 3 ahead of optical disks (160MB/s for Ultrium 6 vs 54MB/s for Blu-ray 12x).

I wonder what happened to the idea of multiple optical read heads. When CD-ROM drives were trying to maintain 52x read rates, I think Pioneer or someone made a drive that read at twice the rate using 2 separate optical assemblies.

Kenwood had a 72x drive that used 7 lasers! http://www.pcstats.com/articleview.cfm?articleid=339&page=2

Citation please!

BD-R's are metal-based, not dye-based. They may be more fragile than DVDs, but they are very light- and heat-resistant.

BD-RE advertised 10,000 overwrite cycles http://docs-europe.electrocomponents.com/webdocs/0bdc/090076...

LTO-5 gets only 200 full-capacity writes http://support2.imation.com/downloads/imn/LTO/Usage_Life_Ima...

Thanks! I was more wondering about the lifespan of a resting medium though.

Yeah, I'm having trouble finding numbers for tape (my google-fu is weak). This page advertises 50 years for BD-R http://panasonic.net/avc/blu-ray_disc/archive.html

Edit: This one says 30 years for LTO-5 http://www8.hp.com/us/en/products/storage-media/product-deta...

Very interesting, though I'm really missing information on what kind of material they are using. While it's true that CD's can take a bit of the kind of environmental abuse they describe (you scratching your game's CD doesn't fall into this category), we all know that in reality they weren't that durable – though I suppose that stemmed more from the fact that most CD's were produced cheaply, corners cut, a lot of abuse by the end user and whatnot. Did you know that there's even a fungus that really likes to eat CD's? ( http://en.wikipedia.org/wiki/Geotrichum_candidum – Not so much for damp archive cellars, then! ).

In conclusion; I think that just enforcing high manufacturing standards won't suffice, there must be some material definition as well. Looking forward for more information on this.

Wasn't the problem with CDs that they had a plastic layer, and a scratchable metal data layer with no protection on top?

Yah, everyone assumed that the clear side was the fragile side and protected it, but actually the label side is much more fragile - and unlike the clear side, unfixable if it gets damaged.

But what happened to the promised sapphire (alumina) coating on disks that makes them scratch proof?

Not exactly; the specification called for a plastic layer on both sides of the metal data layer. In cheaply produced media however the top layer (top here = the one with the print on it) was made as thin as possible, or even omitted and just coated with a printing. Also, there were cases where the two layers were joined/glued together inadequately, not properly sealing the metal layer from the environment.

All of this led to the oxidation of the metal layer; rendering it unusable.

Stamped CDs have nothing but some thin protective layers on top of the metal. CD-Rs, at least good ones, have more protection on top and aren't as fragile.

1TB doesn't take so long to upload nowadays. Isn't this product going to have battle with a new era where we just upload to a storage provider, the provider aggregates customers' stored data on a massive scale and so the underlying storage technology no longer matters?

I'm thinking about services like Amazon S3 and Glacier here, together with whatever competition appears. I presume that at this level, what matters is exabyte level storage hardware (perhaps a robot-managed room of archival discs or tapes). One where the provider can switch technology every few years, and customers never have to notice.

At capacities on the level of terabytes, we're getting to the point where upload bandwidth isn't so much of a bottleneck any more, aren't we?

What? I don't know where you live, but I assure you that there are plenty of places where uploading a TB takes a lot of times (for example, it would take about 3 months with my home connection). Even using the average global upload speed (7.6Mbps[1]) it would take 12 days.

So no, I don't think this is going to battle storage providers.

[1] http://www.netindex.com/

I claim that people who have the speed of your home connection is not the target market. You may want one of these, but this kind of demand will not be enough to support the existence of this product.

Businesses have a choice: they can invest in buying storage hardware for a bunch of their computers every few years, or they can invest in a fast-enough Internet connection and outsource the storage. That Internet connection will bring them additional benefits over just storage.

I claim that the economics are switching in favor of the Internet option, and will continue to do so over the next decade.

> I claim that people who have the speed of your home connection is not the target market.

You know that you are talking about the typical speed of many industrialized countries, like Germany, Spain, France? I doubt those whole countries are not the target market.

Uh, he pointed out the average. For some of us first world countries where the average is even less it's even harder. That also fails to consider places that have quota limits as well as low-bandwidth connections, for example whereby I get 150GB/month combined down & up.

The economics maybe, but the political will is another story. The business that I work for will not even contemplate cloud based anything.

We use expensive 500gb tapes to take weekly backups of our entire site which are kept for 1 year. I'm not sure how many times the tapes can be overwritten but there would be a price-point for these archive disks that would make the tapes obsolete. It would also save a lot of space and therefore money as we are charged to store the bulky tapes off site.

The problem is that businesses don't always have a choice. Maybe big corporations do, but for small companies, you have to deal with that the locals ISPs give you. Where I live (Italy), unless you happen to live in a major city (and even there, the coverage is small and the best speed you can get is 10mb/s) the best you can get is a SHDSL line, which gives you at most 8mb/s for about 300-400€/month. And that's if you're lucky to have a central nearby that supports it, otherwise ADSL2+ it is (1mb/s). I'm sure this is the situation in most of the world.

There's is also the "political" problem (as outlined by everyone else), I can't see every company wanting to outsource the storage of their secret data. Then you're also at the mercy of Amazon, that for whatever reason can stop providing you service (It's a tiny possibility, but it's still there).

For the "Archive Disc" to be viable, it needs to still be viable several years from now. Despite areas with poor connectivity, connection bandwidth will only ever go up (and prices down), and it does so very quickly.

I know plenty of relatively small businesses in my area that use business-class asymmetric DSL connections with an upload speed that tops out under 40KiB/s. Many of these businesses are not in an area laid with fiber-optic cable. I have no idea how much it would cost to get dedicated fiber to a business in an area where it hasn't been pre-laid, but I imagine a nontrivial sum.

Small businesses still generate data that needs to be archived. Uploading a TiB of data at ~40KiB/s would take the best part of a year.

The economics are certainly moving, but I think it will be a long time before on-site storage is obsolete.

A reliable fast connection is still very expensive, and there are several other considerations. When we looked at offsite backup a major consideration was restore times. Even if you can do differential backups, in a disaster you might need to get everything back ASAP and even the best internet connection can't get near the throughput of onsite disks/tapes.

We have a company that employs 120 people. The fastest connection that we can get is 4Mbps down/1Mbps up.

Large companies are hesitant to upload backups of important and often secret information to cloud providers, even if it is encrypted (because who says their encryption will be safe in 20 years?).

This technology is obviously targeting this kind of customer, and I expect it to do rather well in that niche. Providing a cloud-like backup infrastructure internally is a significant cost, is rather hard to get right and requires a lot of maintenance over time.

Even if you have enough upload speed, you have to account for Amazon failure (war, governmental decision, aliens etc) so I would use both cloud and optical media.

There's a lot of people who still suffer with bandwidth caps. I have 3TB stored on Crashplan but I still get nervous about that. Companies have disasters or go out of business all the time.... I'm considering moving to bluray + bank safety deposit box. Depending how expensive these discs are, they may be an option also.

Try uploading 1TB with a bandwidth cap.

Maybe their target customer IS Amazon Glacier. Their competition is tape systems.

> At capacities on the level of terabytes, we're getting to the point where upload bandwidth isn't so much of a bottleneck any more, aren't we?

Depends on the from and to. 1TB at a sustained 100mbs is roughly 24 hours. Or 10 days at 10mbs sustained. If your data is actually written to disk somewhere, I doubt you can sustain more than 200mbps for most services?

So, if you record home video in 4k -- you're going to reach a practical bw ceiling pretty soon. In general disk usage is going up faster than bandwidth -- so archive media isn't likely to go anywhere.

As for Amazon Glacier, cost will come to 120-133 USD per Terabyte/year -- with an additional cost of ~300 to retrieve a Terabyte in 24 hours. For comparison, a 4TB usb3 external drive costs about 150 USD now -- and you'd break even buying 3 each year -- in terms of cost/TB (assuming you could store them for free, and spent the 300 on retrieval as needed).

Your Glacier comparisons should include the fact that Glacier is an actively managed service with very high durability[1] – if you're buying external USB drives, you need at least two drives simply to avoid single-device failures and that's before you get to geographic separation and the software + staffing to provide regular active testing for bit-rot, not to mention some sort of software scheme to prevent data from being altered after it's been written (malware, human error, etc.)

You can definitely beat AWS on price but there's a non-trivial up front investment required unless you're comfortable skimping on the operational benefits.

1. http://aws.amazon.com/glacier/ promises 99.999999999%

Of course. But you get three TB for the price of one on glacier (not surprising, considering they guarantee redundancy) with money to spare - per year of storage. So if you can archive to disk(s) and assume the disks hold for five years there's a pretty big difference in price (obviously, glacier probably will become cheaper in the years to come, but that's still a pretty big difference in the consumer spectrum -- for a business -- probably not so much).

It's definitely a significant savings if your time is free. I'd question how big the savings looks if you assumed your spare time was worth, say, $25/hour.

It seems this is mainly just a larger capacity (1TB) optical disc, and the 'archival' is just marketing?

For that matter, if they were really marketing at those who are professionally concerned with long-term reliable storage (archivists), even the marketing would include some information on what makes this new media any more reliable over the long-term than existing optical media. The press release includes nothing on this, odd for something branded as 'archival'.

It looks like it's just a larger capacity optical disc (which I'm sure there's a market for), with the 'archival' part just being marketting (odd; apparently they think there's a market for this too, even when it's just spin).

CD's don't last forever (http://en.wikipedia.org/wiki/CD-R#Lifespan), so you can't use them for archival purposes. For now, there is no way to tell whether these will be any better.

I believe you can use them for archival purposes if you buy quality discs. Which for me means Made in Japan Taiyo Yuden, although perhaps Mitsui/MAM-A are good. See https://news.ycombinator.com/edit?id=7372810 for my most recent experiences with these discs.

This is really, really, dangerous advice unless you're following proper archival practice and have made multiple copies which are regularly checked for bit errors. For all media, but particularly for CD-Rs, you have to be very careful about environmental conditions – a few degrees temperature difference makes a difference for how fast the dye degrades and unless you actually monitor this, it's easy to have that problem without noticing it until things start failing.

For a small user, you have the extra concern of betting large on a single production run from a single vendor. If you care about archival, you'd really want to burn multiple copies using discs from different production runs and, preferably, different manufacturers.

(This is, of course, why I'd really recommend using CD-Rs only in conjunction with a different technology such as AWS Glacier which has a completely different set of failure modes)

There's actually a lot less commonality in a single production run than you'd think, or so I believe having analyzed Taiyo Yuden disc hub numbers. Different physical machines, I suspect. But of course the "chemistry" will be roughly the same, and thus a whole batch or more could go bad early. Although one might hope they do accelerated aging tests on their production.

You are of course correct about the precautions that have to be used, but there's screw cases for everything you can use to back up your data. I was merely responding to the assertion that CD-Rs are entirely unsuited for archival purposes, and backed it up with my own experiences over a decade and a half.

Me, I live a hair's breath from Tornado Alley, and use multiple means to backup my data, local discs, disks in another room (which got trashed enough to be unreliable in a tornado http://www.ancell-ent.com/1715_Rex_Ave_127B_Joplin/images/) which I've switched to LTO-4 tape, some of which live in a safe deposit box, and the really important stuff offsite to rsync.net (which saved the only important data totally lost locally from the tornado, or at least without $$$ to a data recovery firm). And of course the CD-Rs, which at some point I'll start putting into the disk and LTO bases system.

> Although one might hope they do accelerated aging tests on their production

The experienced storage admins I know share that hope but don't trust vendors not to get it wrong. It's just too easy to miss a factor which turns out to matter.

> I was merely responding to the assertion that CD-Rs are entirely unsuited for archival purposes, and backed it up with my own experiences over a decade and a half.

Question: have you done bit-level checksum validation on that old media or is that just the ability to read without errors? There's a little bit of error correction built into the format but I wouldn't trust it for anything important.

Bit level CRC-32 checksum validation, absolutely. (Which is actually probably marginal for the file sizes concerned, but that's the standard everyone uses.)

However my words are coming across in this discussion, I'm really not a very trusting guy! Many of my early computer experiences were in the early '80s with surplus hardware from the '70s, including a PDP-11/45 that was a bit beyond the 300th DEC manufactured. Of course everyone used magtape back then for backups, and those were quite reliable (originally intended to be reliable off-line storage).

ADDED: and there are other precautions to take. E.g. I only bought my optical media in April and October, to minimize the environmental stress during shipping.

> Bit level CRC-32 checksum validation, absolutely. (Which is actually probably marginal for the file sizes concerned, but that's the standard everyone uses.)

In the digital preservation community, the standard is at least MD-5 / SHA-1 and most people are moving to SHA-256/512. With a CRC-32 check you're likely to get false-negatives for modern data volumes and there are disturbing reports of CRC failures at higher than expected rates[1] which suggest that the best answer is using multiple, cryptographic hashes particularly since the computation has effectively been free for a least a decade except on unusually CPU-starved storage hosts.

1. I don't recall the paper but I believe it was a followup to http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigco...

Well, no media lasts forever, but we still have to store digital content somewhere for archival purposes. (ideally more than one copy, naturally, routinely checked for data integrity).

But yeah, the long-term reliability of media is often a key consideration in archival storage, which is why I think it's suspicious that they brand this media as 'archival' without providing any information at all about what they think it's long-term reliability is, compared to other (optical or other) media.

"Archival" may also mean "non-consumer" as in "don't ever touch the media with your hands" or "don't even think of having a single copy of anything". The physical tolerances of this are so beyond already fragile Blu Ray media I can only imagine these disks would come encased in airtight cartridges to be manipulated by exquisitely precise robots and read and written within very delicate drives.

You think the media is going to be LESS reliable for long-term preservation than existing optical media?

Wow, if so, it sure seems sketchy to brand it as 'archival'.

I imagine these disks would not survive the kind of mishandling DVDs and BDs survive every day. If it's archive, it should be treated so. Also, having multiple copies makes recovery comparatively faster.

In this context, long term durability has more to do with chemical attack from the atmosphere than being handled by human hands.

Having media that lasts for 100 years is little comfort either. What good would do floppies that last for 100 years when the only computer capable of using them here in my house is an Amiga 600?

Huh, so to you, you assume that something labelled "archival" means "is especially fragile, must be treated with care", that's what "archival" and "treated so" means to you?

Odd. To me, something labelled "archival" means "is expected to last a really long time, and be especially robust."

I _think_ that's what it means to most people. Although it's hard to say. But compare to wikipedia says "Archival paper is an especially permanent, durable acid-free paper." Especially durable and permanent, not especially fragile and needs to be treated with unusual care!

If they are marketting something as "archival", I am reasonably certain that _many_ people are going to think that means "especially durable and permanent", not "especially fragile, must be treated with care."

But yes, there are many more factors to long-term preservation of digital content than simply media reliability, including inter-operability with future hardward/software. Integrity of the bits on your media is certainly one factor though. And yes, multiple copies and regular integrity checks is crucial, not just counting on your media to last. Media long-term reliability still matters though.

(I work in libraries, and I work with and next to people concerned with digital preservation, although I don't work on long-term digital preservation myself).

> Huh, so to you, you assume that something labelled "archival" means "is especially fragile, must be treated with care", that's what "archival" and "treated so" means to you?

No. I just suppose that packing that much information so tightly, encoded in such small features on a plastic disc, will make the recording more fragile. It's like printing all your text with a 3-point font. You can print a lot of text on a single page, but any damage to the page may make large parts unreadable.

As for long term preservation, I'd never suggest keeping your data on the same medium for any significantly long time. And never, ever, under no circumstances, find yourself with a single copy of anything.

Sounds interesting. Hopefully they will release more technical information about what makes it "archival" quality. Seems like a normal Blu-ray at first glance.

Sounds like they have a disk with a large storage capacity and perhaps a slower read time and the marketing just did it's self.

I would have imagined some special material that can last longer and is at lower risk from scratches...

like it says, higher capacity (300GB, 500GB, and 1TB) along with higher redundancy.

But they don't say how well it does, compared to the problem of cd/dvd, that fail to work after 10 years for no apparent reason( no visible physical damage ).

>development of a standard for professional-use next-generation optical discs -- > dust-resistance and water-resistance, and can also withstand changes in temperature and humidity when stored.

So it's a new standard with tougher physical requirements. Actually in a japanese press release [1] they present at the end the current archival solution offered by Sony, which consist of a set of 12 optical discs in a cartridge. I'd image this new disc standard could be sealed as well for better protection.

[1] http://www.sony.co.jp/SonyInfo/News/Press/201307/13-0729/

I, for one, am glad that disc is not dead.

I'd bet this will take years to produce in reasonable price levels and by then the capacity will have been made irrelevant by the market.

If these still come as flimsy plastic fully exposed to the elements and can't be touched on the face, lest the data be destroyed, I'll be sorely disappointed.

Seriously, why hasn't anyone made diskette-style discs a common standard? I have never seen a cd last more than a few years, or one single touch with a finger on the shiny side.

For the same reason that the most common way of storing information before the CD was a disc where you scratched waveforms directly into the vinyl: a plastic disc with a thin film on top is super-simple and cheap to mass produce.

Yes, but it's bad for archiving data. My general experience with CDs and DVDs is that two out of every three you burn will be unreadable the moment you burn them and the third one will be readable exactly ten times before it stops being readable, too.

That was alright when I just needed to move data around, or wanted to watch a movie once or twice, but I can't use something that unreliable for archiving.

> My general experience with CDs and DVDs is that two out of every three you burn will be unreadable the moment you burn them

Just a few weeks ago I actually took my old spindles of CD-Rs and DVD-Rs out of storage and moved into Amazon Glacier. The disks had dates from 1999-2007. My results were that out of 80 or so discs, 5 of them had unrecoverable errors (thank god for dd_rescue).

These were burned in my youth, so the method I picked disc brands was "the cheapest disks that aren't those uncoated foil noname ones"

If you're seeing a 66% failure rate, may I suggest replacing your faulty burner or reader? I've yet to experience an instantly unreadable CD/DVD that I've burned, whether using the burned disc in the same drive or another system's drive.

Sounds like you were using bad media, at least for CD-Rs (DVD recordable media pushes the technology too far and is inherently less reliable). Unless they're Made in Japan Taiyo Yuden CD-Rs, which cost around 25 cents per disk last time I bought a bunch a few years ago, or maybe Mitsui/MAM-A media, I would expect those sorts of problems. For DVD recordable media, nothing but Mitsui/MAM-A Gold is worth trying.

I've never had an unrecoverable error with Taiyo Yuden CD-Rs. I just recently pulled 8 out of cold storage with no problems aside from one marginal DVD-R drive, that were burned from 7/13/04 to various dates in 2008. If you want I can check from earlier in the decade, although my first batch of discs were Kodak Gold. I systematically burned and carefully stored about 4,000 during the last decade so I've got a lot to choose from ^_^.

Wait, so you had no problems at all, except for the problems you did have, which apparently work out to something like a 12% failure rate over 6-10 years?

That's not really a great endorsement.

No. Right now I have one marginal Sony Optiarc AD-7260S-0B DVD drive bought back in August 2011. By simply not using that drive, e.g. a 2007 Pioneer DVD drive, or a new "Samsung", my media has always read fine.

OK, the blanket "I've never had an unrecoverable error" is not true if include "on every drive I've tried them on", but then again this is the only drive that's ever started failing on me before it was replaced in the normal course of events. So my success rate in getting my data back without serious recovery efforts, over 14 years minus 5 months, is 100%. But I should go check some of those oldest disks again, it's been a year or two since I've retrieved stuff from the very early oughts.

I don't consider this to be a big thing, given these are commodity consumer drives which will continue to be manufactured to read CDs for the foreseeable future, especially compared to tape, where bad drives are known to eat them and they don't have one old format frozen in time (granted, CD-Rs are less reflective than pressed CDs)....

Well, if you're going to go to a cartridge style format, the competition is hard disks, which can store three or four terabytes each in a highly reliable form; sure, they cost more per disc, but the upfront costs are very low; I suspect the volume at which cartridges would become cheaper is larger than the market would support.

Hard disks are also not very good at long term storage, I believe most aren't designed to hold their magnetic charge for decades without being powered and rewritten.

Sure, but the same is true of every high-density medium. You have to either accept data storage is a dynamic process of repeated copying, or else do it the old-fashioned way and store your data on acid-free paper.

(Well, unless you want to use one of the exotic technologies that etch your data on metal plates or whatever; but in practice, paper is a lot cheaper.)

> I have never seen a cd last more than a few years, or one single touch with a finger on the shiny side.

On the other hand, I've yet to have a Blu-ray fail[0], despite treating some of them rather poorly. This is just with factory-pressed discs though, I have yet to burn one. Has anyone else here had more experience with BD-Rs?

[0]which use more advanced coating such as Durabis http://en.wikipedia.org/wiki/Durabis

And why not save it on a External HDD ? I thought that CDs/DVDs are bad for backup because they crumble.

Hard disks are also bad for backup because they rely on a magnetic charge being retained on the platters. Eventually, this charge becomes to weak to be read reliably, so it is unsuitable for long-term storage. Pressed disks (as in, those not burnt with a consumer drive) can last a lifetime if you take proper care of them, and are not as vulnerable to light, heat or humidity as the typical DVD-R.

Thanks for the information.

If i cant afford to press the disks how should i backup my data?

Right now we are using nas systems with a raid setup

LTO tape or Taiyo Yuden CD-Rs are my recommendations and what I personally use at home. And I haven't checked for a few years to see if Taiyo Yuden is still producing good stuff, but if you're paying around 25 cents per disc it probably still is. MAM-A CD-R media is probably a good bet; I don't recommend DVD recordable media, even MAM-A Gold, because it pushed CD-R technology too far.

Per sspiff, you do need to protect optical media from environmental extremes. Tape cartridges are more physically fragile (although I'm sure there's data recovery firms that'll repack the tape into a new cartridge) and also shouldn't get too hot or humid. And ideally are stacked on edge.

To be honest, I haven't got a clue. Affordable, multi-decade storage is still an unsolved problem as far as I know.

A RAID NAS is a good medium-term solution, and perhaps creating offline backups on disconnected drives every 5 years or so can solve long term storage, though it does require maintenance over time (refreshing the drives). If you are not afraid to use cloud storage, you might use those services, but I don't know how long those will be around.

Note that if you're using cheap, big consumer quality disks and RAID 5, if you lose one you're likely to lose the whole array because at least one of the other disks will have unrecoverable read errors during the rebuild. RAID 6 helps, but the math is still frightening.

Granted, I haven't looked for years at what's been done to address this, but based on the state of the art back then I'd only try this with ZFS, which checksums everything it puts to disk, which among other things catches the incredible screw case of the very complicated firmware writing the right data correctly to the wrong location on the disk (!).

Any idea how much it costs to have a disc pressed? For some things of mine it might be worth it.

Harddisks (magnetic storage) are very poor for long-term storage. Especially modern 2TB+ ones. If you saw the raw error rates (that get repaired using checksums) on a typical drive you'd be very afraid.

Thank you for the information.

I'm almost certain that 1TB isn't large enough for 2014, let alone the next 3-5 years.

I'm sure they would sell you more than one ;)

is there any information about the lifespan the disc? how long cant he data be read before the discs start to lose data because of age? 5 years?

Is there a "plug and chug" way to do PAR (PAR2, etc.) segmentation and error correction if one is not using WinRAR?

Ah yes, this must be targeting the ever-popular "people who didn't pay attention to any of Sony's OTHER proprietary formats that have now been utterly abandoned" market.

I've not heard of Sony doing this in the "professional" market, e.g. while AIT/SAIT has been abandoned, Amazon indicates you can still buy tape cartridges. But, yeah, if you went with them right now you're wishing you went with LTO, or have already switched or started.

On the other hand, https://news.ycombinator.com/item?id=7372595 indicates Panasonic is not trustworthy in this area.

Maybe give this 5-10 years and see if has staying power, adoption by others, etc. The LTO ecosystem shows this sort of thing can be done.

The Holographic Versatile Disc format has existed since 2004 and can store up to 6 TB vs Archival Disc format capacity of only 300 GB - 1 TB. Any idea what advantage ADs have over HVDs?

I think the main advantage is that ADs are relatively less vaporware than HVDs [1].

> Standards for 100 GB read-only holographic discs and 200 GB recordable cartridges were published by ECMA in 2007, but no holographic disc product has appeared in the market.

[1] http://en.wikipedia.org/wiki/Holographic_Versatile_Disc

Did they ever manage to make those? They were announced over and over again but I don't think any products ever hit the market. Vaporware.

Mitsubishi ARLEDIA, long-term DVD-R optical media storage are using this longer. I think they started in 2008.

Mitsubishi ARLEDIA, long-term DVD-R optical media storage has this a few years ago. I think they started in 2008. Archival Disc format is not a great news.

How are we today, little sockpuppet?

(Can somebody with more karma please flay, er, flag'im?)

Reply on news.ycombinator.com