New 5D Storage to Offer 10,000x the Density of Blu-ray | Hacker News Reader

New 5D Storage to Offer 10,000x the Density of Blu-ray

tomshardware.com

115 pointsWithinReason135 comments

In case anyone is wondering what they mean by "5D": they mean it's a 3D array of marks, each of which has both size and orientation. The 3D array is three 2D layers.

According to this definition, a DVD or Blu-Ray disc is "4D" because it can have multiple layers each of which stores 1-dimensional data (pit versus no pit). And a book is at least "7D" because it contains a 3D arrangement of marks, and marks can vary in (at least) what letter they are, how large, how bold, and how slanted. (It would be easy to add to that list.)

I would describe this thing as 2.5D storage: it's basically 2-dimensional but with multiple layers.

If it's genuinely 10000x denser than Blu-ray, great! If it's likely to last longer, also great! But there's no need for this "five-dimensional" bullshit.

(It looks to me as if what they have at the moment is not anything like 10000x denser than Blu-ray. I'm not sure it's even 1x denser than Blu-ray at present.)

A better definition then would be: Assuming each dimension is infinite, how many dimensions do you need to uniquely address a bit.

The infinite requirement is there so the answer can't be reduced to "1 dimension" every time (if you had a 10x10 matrix you could just turn it into an array of 100 elements).

Slight nit-pick, but any number of dimensions can be reduced to one, using a pairing function https://en.wikipedia.org/wiki/Pairing_function or a space-filling curve https://en.wikipedia.org/wiki/Space-filling_curve

Right. I don't immediately see how either of those would work for uncountable infinite sets, but it may be possible.

Clearly some other limitations are required to make "dimensions" a useful metric in this context.

> I don't immediately see how either of those would work for uncountable infinite sets, but it may be possible.

We can think of an N-dimensional space as a grid of (hyper)cube "cells":

- Cantor's pairing function traces diagonal lines through the top-right quadrant of a (countable) 2D grid

- We can extend Cantor pairing to the whole (countable) 2D plane by joining the diagonals of each quandrant (they meet at the axes). The resulting line spirals out from the origin, in a diamond shape.

- We can extend the 2D plane to higher dimensions, again by joining diagonals when they meet at the axes. This results in a diamond-shaped "shell" spiralling out from the origin.

The above gives us a 1D line which visits every cell one after another. Next we associate each unit-interval on our line with the cells we're traversing, i.e. the interval between 0 and 1 is associated with the 1st cell; the interval between 1 and 2 is associated with the 2nd cell; and so on. (We could also jump between positive and negative, like [0, 1), [-1, 0), [1, 2), [-2, -1), ... to include the negative half of our 1D line as well)

Finally, we use a space-filling curve, like a Z-curve, to map each interval to the volume of its associated grid cell. This works for uncountable intervals and cell volumes.

The result would be a line which starts at the origin, and gradually spirals outwards. From a distance, the line appears to form N-dimensional diamond-shaped 'shells'. If we look closer, we see each 'shell' has jagged edges, since it's formed of N-dimensional grid cells. If we look even closer, we see that the line traces a z-curve through each grid cell, filling its volume.

The above works for any pairing function and space-filling curve, although I've written about cantor pairing at http://chriswarbo.net/projects/procedural/cantor.html and z-curves at http://chriswarbo.net/projects/procedural/z.html :)

I think once you are talking about "uncountable infinite sets" and different kinds of infinities, the definition has become very niche.

what you're dancing around is linear independence, which is the idea that two functions can't be described as a function of each other, which also means they can be used to specify every other function in the intervening function space. the least number of linearly independent coordinate functions (usually transformed down to unit vectors) is the dimensionality of the system. note that you can have linearly dependent functions/coordinate systems, which this '5d' moniker is akin to.

> what you're dancing around is linear independence

Thanks! That was a "ohhhhhh of course!" moment for me.

At least that's more succinct and I don't see a way to cheat it.

I think Dimension in the data sense of the word, not Dimension in the spatial sense of the word.

Multi-dimension data is a thing, and it goes a lot higher than 4, 5, or 7.

>I think Dimension in the data sense of the word

Where else is that used? At least to me (and the common layperson), the only usage seems to be from people trying to hype up storage mediums.

also, if we're using that definition, does that mean I have a 7D SSD? The NAND cells are arranged on both the planar axis, and stacked (3d NAND). That's 3 dimensions. Each NAND cell also encodes data using various charge levels, which currently tops out at 4 bits. In total that makes for 3 + 4 = 7 dimensions.

It's a math/information science term that was applied in physics (its most well known usage), not the other way around...

It's mostly a mathematical term which also applies to pieces of data. See here: https://en.wikipedia.org/wiki/Dimension

    In physics and mathematics, the dimension of a mathematical space (or object) is informally defined as the minimum number of coordinates needed to specify any point within it.

In my physics courses we would routinely talk about 6 dimensional spaces when discussing position-momentum space, also we would sometimes refer to two dimensional objects like a moebius strips as "1-dimensional".

> also, if we're using that definition, does that mean I have a 7D SSD?

Maybe. You can use whatever definitions you want if you feel it's helpful. However, based on your description, if you're using a single bit as a dimension, you should figure out how many bits the the dimensional information encodes. As it stands, it's not clear what you're counting as a 'dimension' in this context. Also, normally we think of degrees of freedom as being independent of each other (hence the claim to 5d in this example). The charge information would normally be thought of as a single degree of freedom, so you'd have a 4d nand disk.

One reason it is helpful to think of "a dimension" as an independent degree of freedom, is that it becomes a parameter you can focus on improving. So if you say "Well what can we change, or make more precise?", the answer for the nand case is "Well, we can't easily add another spatial degree of freedom, but we can improve our stacking to improve how much we can pack into the vertical dimesion (but we're limited by Job's obsession with thinness). We can improve our measurements of spatial resolution, which will affect 3 of our dimensions, but not the charge. We can also improve our charge resolution. Let's figure out which one is cheapest to scale."

Now granted, sometimes tweaking one parameter affects the other, so they aren't strictly independent. You can pack a lot of charge on a NAND, but as the other dimensions get smaller, you start having increased difficulty with leaking.

Anyhow, I think there's more here than you're giving them credit for, even if they are guilty of tooting their own horn a bit (shocking).

But yes, the article presented here would be greatly improved if it gave examples of how to classify existing technology using the researcher's dimensional classification scheme. It's a common scientific writing tactic that I'm surprised was not used here.

That is still O(N^3) in my opinion. If you scale up the media in all directions, your data scales like the cubic. The charge level makes it maybe 3.1 dimensions, but it isn't really scalable.

Now if you could add arbitrary voltage levels without losing precision, that would be 4D.

Which reduces to one single dimension anyway, on a storage medium: my bytes are stored from offset 0 to offset N.

That's a bit reductive.

Maybe for some. For most uses though, people just want to write data to and read it from the storage media. They don't want to think about the coordinates their data was written to on the DVD.

Ba dum tssss

The fact that spaces of different dimensionality can be put into a bijective map does not mean that dimensionality is not real or useful.

Yes, the dimensions are an abstraction to make it easier for our human brains to organize certain collections of data. I don't think it makes them any less valid, otherwise we would refer to every data type as an array of bits and nothing more.

Isn't this normally referred to as "degrees of freedom" rather than dimensions?

Statisticians and mathematicians call them dimensions, machine learning engineers call them "features". Statisticians and mathematicians usually interpret observations as vectors in n-dimensional space, which leads to some interesting geometric results. Hence the term dimension, we do mathematics with them as if they were actual points in some high-dimensional space. Degrees of freedom are something a bit more nuanced that are of use when you start to do specific statistical tests, and they do not have to be exactly equal to your amount of dimensions.

No, these are separate concepts. As an example: a simple linear regression through 7 data points has 2 dimensions (x and y) but 5 degrees of freedom (number of data points - number of estimated parameters).

Sure, "5D" _could_ make sense. It just happens not to in this case.

They're mixing up two different sorts of dimensionality that it doesn't make any sense to mix up.

1. How many parameters does it take to describe where you're looking for a given piece of information?

This is (kinda) a measure of how many pieces of information there are. Increasing the number of dimensions can mean a very large increase in the amount of data -- imagine going from a 1000x1000 array of things to a 1000x1000x1000 array.

The number of dimensions in this case is "two and a bit", because they're using a small number of 2-dimensional layers.

2. How many parameters does it take to describe what you find when you look at a particular piece of information?

This is (kinda) a measure of how much information there is in each thing. Increasing the number of dimensions here typically means a rather small increase in the amount of data. E.g., the difference between "1D" and "2D" here would, in the simplest case, be a factor of 2.

The number of dimensions in this case is 2, because in each voxel there's a thing parameterized by size and orientation. (I think.)

So, three dimensions of the first kind and two of the second. Add 'em up to get 5D, right?

Wrong. (At least, if your aim is to inform rather than to deceive.)

First problem: these two kinds of dimension do not behave in the same way. Suppose there are N positions along each dimension, and you have A type-1 dimensions and B type-2 dimensions. Then you have N^A places to look for data, and each one stores B log N bits. So adding 1 to A multiplies the amount of data stored by N, typically a very big change; but adding 1 to B multiplies it by (B+1)/B, typically a rather small change.

Second problem: the "baseline" is not zero type-2 dimensions but one. You always have at least one type-2 dimension, or else you aren't actually storing any data. So e.g. if you have a grid of memory cells, that's 2 type-1 dimensions and 1 type-2 dimension. Are we going to call this 3D storage? I really hope not.

Two-and-a-bit-th problem: as mentioned above, their third type-1 dimension is really hardly there: they have three layers. I guess that is a third dimension, kinda, but it's pretty underwhelming.

This is 2-and-a-bit-dimensional storage, with each storage location being able to hold a bit more data because of this size-and-orientation thing. It sounds like it's an impressive technical achievement and might turn out to be a great storage medium. It just isn't in any useful sense five-dimensional.

I said at the start that "5D" could make sense. How? Well, suppose you have a system where you have things arranged in a 3D lattice, and then to query each thing you shine light at it and see what it does. And suppose you're able to make each of your thing store different things for different wavelengths of light, and also different things depending on what angle you shine the light at. In that case, the number of type-1 dimensions might be as large as 6: storage locations are indexed by three spatial dimensions, two dimensions describing the direction of the light beam, and one dimension describing the light wavelength. (If you had a system like this, then I bet that in practice those last three dimensions would all be "small", like the third spatial dimension in this silica-glass storage.)

Counting size and orientation as extra dimensions would be justifiable if you could overlap dots of different sizes or orientations, even a little.

You still need 5 values to retrieve a single bit of data, even if only two of those numbers have more than 3 bits.

They mentioned 500TB for a single glass disk, isn't that 100,000x more than a 5GB blu-ray, or are they comparing it with the 10-layer blu-rays that never really hit the market?

Is this the same UK group that did all the holographic disc research that is basically vaporware in the mass market (I don't want to call their research vaporware, just that it never hit the consumer market)?

Edit: Whoops. 4.7 geebees is for a regular DVD, not a Blu-ray. Dual layer blu-ray is 50GB

They did indeed mention 500TB. That doesn't mean they have actually stored 500TB anywhere, or that they have a system that's currently capable of storing 500TB anywhere, or even that they genuinely have good reason to think that their system can practically be made to store 500TB in something the size of a Blu-Ray disc.

Maybe it can! But so far as I can tell the 10000x figure is much more what they hope might be achievable than what they are in any position to do.

I don't have access to their actual paper. But the reports in the popular press seem to indicate that the individual voxels are on the order of 500x50nm in size, compared with Blu-Ray pits which are about 500x500nm. So that's a density increase of 10x within each layer. Each voxel apparently stores 4 bits, compared with one bit for a Blu-Ray pit, so that's a density increase of 40x.

So if they're claiming 10000x, I think most of that increase has to be from having many many layers. They'd need 250 layers. So far they have done 3. It seems to me that the technical challenges in increasing the number of layers so greatly without reducing the density of voxels might be very great, because to read or write a voxel you need to point a laser at it, and as the number of layers increases they start getting in one another's way.

Maybe they already have some ingenious solution to that problem -- again, I don't have access to their actual article. But I tend to think that if they'd solved that problem then (1) they would be making sure that the articles in the popular press tell the world of that achievement and (2) they would have demonstrated it with more than 3 layers.

[EDITED to add:] Oh, wait, at least one of their publications is available to read: https://www.osapublishing.org/optica/fulltext.cfm?uri=optica... This one talks about "50 layers", though I'm not 100% sure whether they mean 50 full two-dimensional layers stacked on top of one another (their diagrams seem to suggest not), and says that with the parameters they used the amount they could fit on one disc would be ... 1.64TB. That's a ~30x increase over a dual-layer Blu-ray disc. Not to be sneezed at, for sure, but it's a lot less than 10000x.

That paper is entitled "High speed ultrafast laser anisotropic nanostructuring by energy deposition control via near-field enhancement". It seems there's another one called "5D optical data storage with 100% readout accuracy in silica glass". Same authors, approximately same publication date, so it seems a bit unlikely that one of them is claiming 300x denser data storage than the other, but who knows?

Blah, seems a long way from usefulness.

I guess it comes down to areal density vs hard drives. If we want a miracle data storage technology for backup, it would fundamentally need to either be much better in bits/in^2 or a lot of layers.

My google math says 50nm x 50nm x 4 bits is right at a terabit/sq inch.

HDDs are around 1.34 terabit per sq inch. They have more tricks, but it does seem that it'll be harder to move forward.

I guess a bluray was 12.6 gigabits/sq in, and could at least be stacked to four layers with dual-sided dual-layer, about 50 gigabits/sq inch. I think theoretically optical media is very durable, but I still remember the organic dyes in consumer DVD-Rs fading pretty quickly.

Hm, SSDs are passing hard drives in areal density. A 2016 article says they hit 1.6 terabits/sq in. I think SSD makers have closed in on over 64 layers/stacks, and they are at 4bits/cell now. Oh jeez, are they doing 6 ad 8 per cell too? Yeesh that seems very unreliable.

SSDs can't shrink their cells because it reduces durability of the cell, so their path forward is more stacking or just stacking chips on chips as fabs making them drop the costs. SSDs also fail at long-term storage.

So maybe the mythical 500TB cube of data will still have to be optical/holographic.

> But there's no need for this "five-dimensional" bullshit.

Sure there is. It's going to sell great with idiots. I still remember morons super hyped about Opteron trying to claim it was an optical CPU.

I don't know, in my book the idiots are the big brains that thought that calling it 5D would be a smashing idea. Can't wait for the next version being sold as 6D+

It's probably the same type of people that decide the USB technology names (SuperSpeed, UltraSpeed, HumoungousSpeed) that are hard to follow and understand even for hardcore nerds like us.

Marketing would be more appreciated if it didn't take laymen for morons, as you call them.

Yes, the same ones that keep bumping version numbers absurdly. They do it because it works. It doesn't assume all laymen are morons; the people that know it's value will buy it anyway.

The other day I was talking with a friend about the high cost and perceived fragility of tape drives for long term personal archival while bemoaning the loss of higher density optical storage technology. It's not easy to backup or shuffle around terabytes of personal video, audio and pictures. Hard drives aren't exactly the right media for safe deposit boxes.

In the 90's I had an Iomega Ditto 250MB drive that held 120MB per tape for $15-20 per tape and my hard drive was 540MB. This was when hard disc storage cost $1000/GB. The tape drive had a quarter the capacity of the hard disc and was cheap by comparison. It was practical. Later when CD burners came out they offered 650 then 700MB when hard drives were less than 10GB. Then they became too small and DVD came out but shortly was also passed up as 4.7GB wasn't much storage. Bluray was DoA with too little storage to be of real interest with a paltry 25-50GB.

I'd love to see CHEAP >=1TB optical discs with a write speed of 100MB/s allowing us to burn a disc in ~10 minutes. I'd happily pay $1 a disc in bulk and upward of $1000 for a drive. I'd also happily pay $10+ for an archival disc that has a guaranteed shelf life of 50+ years. Bring back the 100 disc juke boxes while we're at it ;-) Cheap storage for everyone!

Some have theorized that this is what's behind Amazon Glacier, and that they want to keep it to themselves: https://storagemojo.com/2014/04/25/amazons-glacier-secret-bd...

I also crave for such cheap storage, perfect for archival/backups, especially when combined with modern snapshotting.

HVDs never really materialized, despite being promising, and actually on the market at some point, IIRC: https://en.wikipedia.org/wiki/Holographic_Versatile_Disc

A lot of other medium were developped, never commercialized: https://en.wikipedia.org/wiki/3D_optical_data_storage

A new optical storage medium is long overdue. This time, it will probably be without support from the media industry.

I'd take an on-prem solution for cheap archival storage any day over Glacier.

With egress costs on s3 that may not be viable if you generate data on the cloud

Is the suspect for Glacier proven?

I can't see them using optical based on their price structure (compared to bluray media costs per GB). And I doubt they own a manufacturing facility to crank out super-secret multi-terabyte-sized optical disks. However it fits right in with the cost of LTO tapes. Quick back of the envelop: Their web site says 1$ per TB per month starting cost. That is $12/TB per year. A 12 TB LTO-8 tape costs around $70 retail, which puts it at $5.60 per TB. So the cost of the media used to store customer data is paid back in 6 months, or slightly more if they are using parity tapes.

That would be my guess, the customer data hits disk first, then gets enough parity and split among enough data centers to hit their reliability target.

> A new optical storage medium is long overdue. This time, it will probably be without support from the media industry.

If anything, I hope that will massively help its adoption. Blu-Ray is a DRM cess-hole of unnecessary complexity. Players – which many people bought blu-ray drives to be – have to be cryptographically secure, contain "upgrade only" firmware (to enforce key revocation mechanisms) and contain a fairly complex microprocessor. There's no official linux support.

All of these anti-features directly compete with "low cost" and "widely available". If I could put 10 TB on a DVD-sized disc, even as a WORM storage medium, and the disc was cheap, I'd buy the thing tomorrow

Why are tape drives fragile? I use them for my long term backups of personal files I thought they were robust ?

What scale are you working at and how have you made it cost-effective versus a stack of external HDDs?

I am not that poster, but I also use tapes.

One may save about $14 per TB by using tapes instead of external HDDs (which strangely have become cheaper than internal HDDs).

In order to recover the cost of the tape drive, you need to write about 200 TB.

I have about 50 LTO-7 tapes, i.e. about 300 TB of compressed archives, so I have recovered much more than I have paid on a LTO-7 tape drive, 5 years ago.

Nevertheless, even if would have written only 100 TB, I would still have bought the tape drive, just for the peace of mind, because after the optical discs became too small I have used HDDs for archival storage, but after a few years in cold storage they become unreliable. If I had not stored duplicates for each HDD, I would have lost a lot of data.

Of course, someone who needs to archive just 20 TB or even 40 TB of raw data, can buy just 2 or 4 HDDs and store each file on 2 HDDs and that should be OK.

For 100 TB or more of raw data (i.e. 200 TB, when counting the redundancy, because also for tapes you should store at least 2 copies, preferably not in the same location), tapes become cheaper.

My brother got a cheap LTO tape drive on eBay and he bought a few LTO tapes so I think I bought one off of him for a few dollars more.

It’s only a few hundred gigabytes and the tape holds about one terabytes .

Note this is my long term cold storage I have backups made on my MacBook Air (hard disk) and my sold MacBook Pro (ssd which I will use on my new 2021 MacBook Pro eventually).

I also have backups using blu ray discs and I have one long term storage blu ray that I have been meaning to use

iomega stuff was generally rather precarious. LTO drives are solid.

Optical discs aren't torn apart when the reader malfunctions?

https://forums.tomshardware.com/threads/my-cd-shattered-insi...

I've had a CD-ROM back in the day shatter on me like this. I'm pretty sure its a common enough story that everyone 'knows' it could happen... but was rare enough that not everyone personally came across it... but I never got to the bottom of it (why it happened, or what conditions led to it). It only ever happened to me once after all.

It sounds like x56 speed drives were just too fast for some CD-ROMs. We went from x1 speed in the late 80s to x56 speed by the late 90s, so it makes sense that some CD-ROMs just weren't designed for the speed increase and maybe shatter.

Alternatively: maybe some physical damage existed. We beat up CD-ROMs back in the day: storing them in backpacks without protectors and whatever. They were very reliable in terms of reading, but maybe those kinds of mistakes would lead to fractures that would literally blow up on you later.

Even then: the most common protector were paper-sleeves, because dust / scratches were what we were most worried about.

---------

We used to carry around stronger plastic cases in the early 90s. But by 00s, it was all paper-sleeves. The plastic-cases would themselves shatter when carried around, but the CD-ROMs / DVD-ROMs were stronger, lol.

Never seen it but have seen many ruined tapes over the years.

I remember my x52 drive would make at least 2 or 3 "test" spins every time I'd insert a disk, just to see how fast it can spin it. It felt like an airplane taking off every time I'd put a new scratch-less disk.

I had one year warranty on that drive. This "feature" stopped working exactly one week after my warranty expired, it would just go full speed regardless of how beaten the disks were.

That's how I got two of my heavily scratched disks shatter.

A few more weeks down the road, the function that's supposed to slow down a disk before a full stop, also stopped working... so now even if a disk was spinning at x52 inside and you'd press the eject button, it would instantly stop the motor and eject it!

That's when I finally threw it away and got myself a "normal" x46 or x44 (don't remember) drive...

Optical disks degrade due to bit rot when the substrate of the disc starts to peel off

They are not fragile in an absolute sense, the poster meant that no matter what tape drive you have, it will break down many years before the tapes written with it would become unreadable.

After the tape drive becomes unusable, you may no longer be able to buy another compatible tape drive, because they might be discontinued.

For example I am using LTO-7 tapes. For the moment, I do not have to worry, because the current LTO-8 tape drives are able to read my LTO-7 tapes and the same should be true for the future LTO-9 tape drives.

However, when LTO-9 tape drives will be the norm, I will have to buy a LTO-9 tape drive and many LTO-9 tapes and copy all my archives, because if I would delay this I would risk to no longer find even LTO-9 tape drives. In that case my tape collection will become unreadable, even if the tapes themselves are guaranteed for at least 30 years.

I have already lost many years ago some archives that I had on QIC tape cartridges, because after my tape drive broke I was not able to find any other compatible tape drive.

In my opinion, it's best to adopt industry practices for home life here. Schofield's 2nd Law of computing comes to mind: if your data doesn't exist in two places, then it doesn't exist anywhere.

Offloading reliable backups to companies seems reasonable. It's a lot more money than archiving to physical media, but it's also a lot more reliable. NAS systems also make it easy to have reliable, hot on-site data that keeps a copy of the data backed up to a cloud service. Many of these cloud backup services are also zero-trust.

Nooooooooooo, you're supposed to back it up to the cloud and pay some startup jerk $XX/mo for the privilege! [/s]

Says whom? Most cloud storage have huge free tiers. I have unlimited storage on telegram.

I bet you actually have "unlimited" storage.

Can you name something that is not "unlimited"?

I don't have a limit, I never hit the cap. By that logic nothing is unlimited (texts, calls, data, etc), but realistically it doesn't matter because I don't use it much. I don't care to test it, I use it sparingly and never had a issue with limits.

I assume that my S3 bucket is as close to unlimited as it gets.

Its also "unlimited" isn't it? There is no such thing as unlimted. As a subsciption you will also lose it if you stop paying.

I imagine it's as unlimited as it gets considering the sun is going to explode in some billion of years. However, I have no trust in having "unlimited" storage on a service where I'm not explicitly paying for it.

So will they intake 10TB?

Never tried my upload is low speed and I don’t have that much data.

Telegram upload speed is limited to 3MB/s.

Doesn't matter how high your connection speed is, it is capped from telegram's end.

I have been talking and asking about this for years. And was told about Archival Disc [1] on HN.

500GB per disc right now, and only came out in 2019. 1TB per disc in the future but I suspect not in the next two years.

I wish they have a more consumer, or prosumer friendly version though. The current drive goes for about $9K [2]

There are Archival BluRay disc that offer 50 + year storage, but they are only in ~50GB per disc.

[1] https://panasonic.net/cns/archiver/optical_technology/

[2] https://www.bhphotovideo.com/c/product/1549175-REG/sony_odsd...

I have very little hope for the price for these things to come down to prosumer levels. The best I can hope for is for local libraries to have one available for use (although you need like 8 hours to fill the 5.5 TB maximum and they're so expensive you don't want to lend them out). However, I have very little hope for that as well since most consumers are just told to use the cloud.

>since most consumers are just told to use the cloud.

Yes and I dont like it this trend at all. But it seems we dont have any other way to fight it.

They remind me of m-discs, which seems to be cheaper as well. How is data stored on those Archival Discs?

I burn 100GB backups via blu-ray. One or two is enough for my non-movie files. They started stagnating a few years ago when streaming ascended however. Shame as I'd like higher capacity as well.

Cheap USB flash drives are already moving into this space. I just got a compact 256GB one for $35 or so, just a matter of time. SD cards would probably be more convenient, but they aren't sadly aren't read in devices like stereos any longer.

That much music (even at lossless compression that’s a lifetime of nonstop music) is completely impractical to navigate using a common stereo UI

Not really, it's about five clicks to start an album, which I'm pretty adept with, and plays for an hour. I have several hundred in Letter/Artist/Year,Album order. Less work than futzing with the radio constantly, which is generally terrible (unless you love tired top-40).

Assuming you're referring to the parent comment's 100GB drive (or even the 256 GB one) that's really not true. A quick `du -sh` tells me I have right at 100 GB of FLAC music, and I've easily listened to every single track. There are, in fact, only 3770 tracks. At an average of 4.5 minutes a song (guessing), that's only around 11.8 days of music. About 8 hours a day of music for a month!

Or to calculate it another way, a typical FLAC stereo track might run about 800 kbps or 100 kB/s. 100 GB will let you store 11.6 days of music at that bitrate, which is just about the same total I got for my collection above.

To be fair, the parent said "impractical to navigate using a common stereo UI" which doesn't have much to do with the capaciousness of a library, more about how easy it is to browse/search/find/choose what you want.

Some stereos have pretty small, monochrome displays. For example this seems to get mentioned a lot in the UK, partly because they are advertised through mainstream channels at older listeners: https://www.brennan.co.uk/

Dunno if you call them "common".

Just remember to use those flash drives and re-copy them occasionally, like once every couple years.

The data is supposed to be valid for a decade but it is not a permanent storage system.

The problem is that much of the demand for cheap, shelf-stable backup media has gone away over the last decade or two. Most backup jobs either go to cloud or to disk, not tape. And because of that, tape drives are more expensive than ever, which takes away much of the benefit of the cheap media.

LTO, for example, is relegated to archival use-cases nowadays, such as video production. You'd think that would make sense, because the media's shelf stable... but it's actually really awful for archival. Why? Because new drives can't read old tapes past two generations, and old drives stop getting made. So every time a new tape drive generation comes out, a movie studio can't just upgrade drives in their libraries and use the new tapes. Nor can they just hold onto their old drives forever - what happens when they start breaking, and they can't read their old tapes anymore? (Or worse, an old/failing drive eats a tape?)

Every movie studio with an archival program is on a constant upgrade treadmill, including costly media migrations. That is, they're copying data from old tapes onto new, so they can maintain compatibility with new drives. This entirely defeats the purpose of shelf-stable media; if I can't buy newly manufactured drives for old tape formats then who cares if the tapes themselves can sit in a box for 20 years?

Now, let's say you're not a movie studio with a massive digital archival problem. You just want to backup 5 terabytes. Unfortunately, 5 terabytes is so little that tape vendors have forgotten how to count that low. It's far cheaper (not to mention, more performant) to just buy a bunch of disk drives and migrate data between them. Or just pay Amazon to do it. Which is what everyone wound up doing.

> This entirely defeats the purpose of shelf-stable media; if I can't buy newly manufactured drives for old tape formats then who cares if the tapes themselves can sit in a box for 20 years?

Why can't you buy new drives for old tape formats? What new features are these new formats providing?

The new formats have higher storage density than the old ones. First generation LTO (LTO-1) stored 100GB on a tape: how much would you pay for a drive like that today? Unsurprisingly, they don't make them any more. LTO drives are read-compatible with tapes from two generations earlier, so you can read LTO-1 tapes on LTO-3 drives, but LTO-3's capacity was 400GB, so no one buys those any more either. The current generation (LTO-8) has 12TB per tape, so you can see the upgrade treadmill chugging along.

https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations

Tape drive backwards compatibility is an issue for sure but optical drives have had decent backwards compatibility so far. DVD drives can read CD's. BR drives can read CD, DVD and BR. I think backwards compatibility is more feasible for optical as the larger pits on older media can still be read by narrower wavelengths. They'll just look like giant craters flashing by instead of nice neat pits.

AFAIK I believe that they actually do use different lasers for different disc types, but the cost to put two or three laser diodes in one drive is basically nil these days.

The big problem with optical storage is the same that tape struggles with - it's primary market shrank.

This was a bit of a self-inflicted wound. CDs and DVDs were basically ubiquitous, drives came down in price dramatically, and the storage densities and pricing was competitive with competing technologies. BD is where that all started breaking down. BD had a format war with HD-DVD, and both formats relied upon new and then-expensive technologies such as blue lasers and quad-layer stacking. PC manufacturers (notably Apple) balked at the cost of either format. Microsoft backed HD-DVD, but wisely decided not to bleed money by not sticking an HD-DVD drive in the Xbox 360. This created an odd situation where the Xbox 360 sold better as a game console, but Blu-Ray outsold HD-DVD because Sony had let the PlayStation division bleed out for a couple of years.

Of course, Microsoft still wanted the 360 to be able to play HD movies, and their response was to partner with Netflix to make streaming a reliable (arguably better) substitute for movies on optical media. In other words, Sony killed HD-DVD, so Microsoft killed the whole home video market by pushing digital distribution for everything.

If you're wondering, there are successor formats to Blu-Ray. Interestingly enough, the whole quad-layer stacking thing was rebranded as BD-XL and UHD-BD (which the PS4 doesn't even support), and there's also a proper successor format for data storage called Archival Disc. Except the latter is basically an LTO competitor - you can't buy bare AD drives and discs and use them like Blu-Rays. The technology only exists in archival systems like Sony's ODA system (Gen 2 and Gen 3; Gen 1 used Blu-Rays), which has discs stacked in cartridges that have to be loaded into special, very expensive readers and media libraries.

I believe streaming killed optical disc for video even if there are no format war. It's just convenient and quality is fine for most people.

To add: LTO speeds are currently in the 300MB/s range: if you want > TB size storage you likely don't want ~100MB/s for regular use. I work on systems where we are regularly writing multiple tapes of 5+ TB and if you get only 100MB/s you're waiting hours and if something fails or gets stuck you can be thrown off schedule by more than a day very quickly. Even at the current 300MB/s speeds it's very slow. If personal archival or even small-scale archival is your interest getting network storage is probably easier, faster and cheaper. What's more, is that you have to keep some kind of inventory of your tapes so you know where to find data if you ever need it. Depending on the size of your collection you can be looking for even more expensive equipment to manage the archive of your archives.

Not to mention that 300MB/s is considered best-case conditions for sequentially stored data. Anything even remotely random is going to have far slower archival times, unless you have archival software that does parallel I/O. (In my experience, not many archivers do this.)

Most production tape installations include dedicated disk storage to act as a buffer for the drives. You archive to disk first, so that your drives aren't tied up, and then write the now-sequential data to tape.

This is all well-and-good, but how do we read all that data in 500 years time, or even 20 years time?

People were busily archiving data to CDs and DVDs two decades ago, thinking 'yeah this digital storage media will last for centuries!'... Loads of bit-rot combined with complete lack of DVD drives still in common [computer] usage, has totally shown that the more complex the storage media the shorter it's useful lifetime is. Compare this to books, which can still be read a millennium later.

Let's not forget actual rot too. I peered into a stack of CD's not long ago to find that a lot of them have quite literally rotted while sitting in a dark and dry closet. No dust, no humidity, no smoking, no abuse, just a spindle of CD's recognizable only by their sharpie'd identifiers covered in pits of rotted foil.

Temperature? I've yet to experience rot on decades old discs at home. Most sit in a basement that doesn't exceed 68*F

A constant 72F, this is an indoor closet, top shelf, no moisture incidents.

It's a well known problem: https://en.wikipedia.org/wiki/Disc_rot

Yup it's a well-known problem, but several of us still have CDs decades old that still do read totally fine. That doesn't mean we rely on them as backups: mine have long since been copied to DVDs, then more recent DVDs but also backup'ed on harddrive / servers etc.

I still have music CDs from 1985 and CD-R from 1995 in perfect condition. None of them have gone bad.

CDs and DVDs are pressed. Recordable media is not and, with the wrong conditions, will rot.

DVD-RWs will get corrupted if you look funny at them.

The linked article also explains:

> DVDs are more resistant to rot > DVDs and CDs use aluminum layers for the reflective 'data encoded' layer > Discs using a gold layer are extremely resistant to corrosion and thus disc rot > Blu-ray discs use a silver-alloy for their reflective layer.

Though not linked, or stated, it is quite likely that the BR discs are significantly less likely to be damaged due to disc rot for similar reasons as gold-layered discs are, though not necessarily as resistant as gold.

Disc rot though appears to stem due to the damage of the lacquer or plastic layer on discs, which if the disc then remains undamaged, it should not be as susceptible to rot.

We are mostly talking about burnt discs which achieve their burning via chemical reactions with light. Those disks rot.

Any standard Blu-ray/DVD/CD you have has no rotting as it's simply pressed.

While the pressed discs should have a longer life, there are some where the metallic mirror oxidizes in time and it becomes partially transparent, which results in reading errors.

The oxidation speed depends on the quality of the lacquer that protects the mirror (on the label side), so it varies greatly between manufacturers. As an end-user, you cannot predict the lifetime, you can just hope for the best.

There have been archival CDs, with gold used for the mirror instead of aluminum. Those were immune to oxidation, but they have been discontinued, due to high cost.

Same - maybe something oxidised.

Regular CD's and DVD's have an organic layer that will rot away. There are alternative styles of discs which use a ceramic layer instead of organic which are rated to last a thousand years. These discs can still be burned and read in most standard burners and readers.

https://www.mdisc.com/

> This is all well-and-good, but how do we read all that data in 500 years time, or even 20 years time?

If the data is important enough, we'll find a way. Perhaps by then camera phones will have sensors with quantum resolution that can capture the optical pits in one snap and use AI or whatever to reconstruct the bits from the image on whatever magical storage exists.

There's no lack of DVD drives, they are widely available on Amazon, BestBuy, eBay.

They're also pretty cheap ($15-30), powered fully over USB—no extra power cable like in the old days—and they're tiny.

Note that for restore purpose, generally slim drives aren't good to read not good quality discs. Better to buy 5-inch bay sized drives.

I think right now the most permanent way to store digital data is have enough redundant copies on enough impermanent media one is likely to survive. For example my CD copy of 1998's HARDWAR might be lost due to disk rot, but there's enough copies of copies of copies floating around on various places it's still easily accessible.

> This is all well-and-good, but how do we read all that data in 500 years time, or even 20 years time?

I know it won't last due to bit-rot but all my CDs burned in the nineties are still readable.

> Loads of bit-rot combined with complete lack of DVD drives still in common [computer] usage...

They are still very common: both in desktop and as USB-connected readers/burners. You can find countless models on Amazon. I don't think they're going away anytime soon.

> People were busily archiving data to CDs and DVDs two decades ago, ...

That's precisely why they're not going away anytime soon.

There are even companies like FB would, at some point, were burning insane amount of data to Blu-Ray discs.

> Compare this to books, which can still be read a millennium later.

Having a book from 1575 here I do agree (not quite a millenium but still). However the solution to disc bit-rot is simple: once every ten years or so, read your discs and burn them on the new fancy tech of the day (say CDs to DVDs, DVDs to Blu-Ray, now Blu-Ray to these 5D thinggies, rinse and repeat).

It's not as if new mediums had less capacity than older ones.

> I know it won't last due to bit-rot but all my CDs burned in the nineties are still readable.

Were these the more expensive archival disks? I lost all ~50 of my old CDs and DVDs to rot, within 10 years. They were middle range Memorex.

What kind of data is even worth storing? If it’s text based it’s not hard and it’s not very large. If it’s media it’s probably some uncompressed videos and pictures that are overly large, phones that take 16mb photos that aren’t going to have more useful information over compressing it to 400kb. I am sick of my phone taking large photos that don’t have any clear benefit to me aside from clogging my phone with unreasonably large files that don’t look better, Apple wants to sell me iCloud by not allowing me to take lower resolution images.

Optical drives aren't common in most consumer devices but that isn't really a hurdle if you're trying to recover archived data on optical media. If you're only worried about DVDs a USB DVD drive can easily be had for ~$20. Blu-Ray drives aren't very expensive, I included one in my last desktop build for the purpose of data archival.

Many consumers figured optical disks would last a long time but many knew about disc rot from the get go. I picked optical media that marketed itself as archive quality. My oldest discs are reaching 20 years old now and I've lost less than 10% of the discs I've burned that were burned on good discs.

These days I burn data to M-DISCs which use a ceramic layer instead of organic. They're rated to 1000 years and the Blu-Rays are quite durable. I've had one bouncing around my desk and outside for several years. The one on my desk has zero errors while the one outside has a few bit errors.

Some guys I know propose a storage on 35mm-film solution that looks pretty foolproof for a couple of centuries at least.

https://digifilm-corp.com/preserve-your-works-for-eternity

To read back the data, you need something to digitize the film (a flatbed scanner would work, I've tried, or probably even a smartphone with a a DIY magnifier), then something to decode the data (the necessary source code is provided in clear on the first images of each roll).

I'm having trouble finding the data density. How many lbs of film/GB?

It's not very dense at the moment, around 600KB per frame IIRC, with lots of redundancy and while staying really easy to read even with DIY hardware. They think they should be able to cram about 2MB per frame without compromising durability and without any compression (but lots of in-picture redundancy).

That should give about 200 GB per roll of film (the large ones).

It all depends on how technology changes. We can still read floppies written in the 70s with essentially a motor, something magnetic and a wire going around it (and a computer reading the currents). I’m pretty sure we’ll know how to cobble together an optical microscope and camera device to read at least one bit out of every volume cell. Let’s just hope the data format is simple enough.

Archiving data is a process, not a one-time effort.

You can't just store data on a medium and hope if that medium will last.

You have to periodically check for defects and from time to time, transfer data to fresh media.

At some point, you may even need to convert the data to keep it readable...

To quote a comment from Reddit: "I can't wait to never hear about this again."

Yep. Still waiting for my tesa drive.

> The new approach can achieve write speeds of 1,000,000 voxels per second, the equivalent of 230 kilobytes of data per second

> 500 TB of data on a small disc

So, it would take 70 years to fill it. By the time you can fill it we will have RAM with higher density :)

> `The researchers say that with improvements to the writing techniques, particularly by taking advantage of parallelism, they can design a system that can fill that same 500 TB in a mere 60 days.`

It's possible to get it up to 100 megs per second, which is write speed of a good hard drive. And I suspect it could be scaled up even more. Progress in this field is quite good, 8 years ago they put 300kb on it, now it's 5gb.

Off topic: if I wanted to digital store files to pass down to my children that might not be access for decades, what's the best medium to store such data?

Floppy/ZipDrive/CD-ROM/Blueray/USB-A are either obsolete or are on their way out. What's a future proof storage option? (and if you said Cloud Storage {X} - just look at how many services have folded in just the last few years alone). Secondly, what's the best file formats also that will be future proof?

I'd hedge my bets and store it multiple ways in multiple formats. Including physical (print off pictures, diaries, etc. on acid free paper with acid free inks)

No such thing. Other poster had a good idea, duplicate everything every ten years. I suspect flash drives will have the best longevity in the medium term however.

As far as formats stick to either very popular or very simple. E.g. for video in 100 years time there is still going to be a way to be able to decode h.264 videos just because of the shear amount of content from this era that was encoded in it. Alternatively use a format that is so simple it doesn't matter if there is a decoder in 1,000,000 years as it's trivial to make. On the simplicity side take PPM or bitmap images for example, one can get a decoder up and running from scratch in a matter of minutes.

The harder half is the storage of the digital information over time. Interfaces, filesystems, formats, and devices are not made to last decades as there is no need for the vast majority of data to sit for decades without being transferred to newer, cheaper, and smaller media. Any solution you come up with to avoid copying will therefore be esoteric and one off, not to mention likely complicated for any significant amount of data, making it hard to guarantee easy enough readability even if it is physically fine in 100 years.

What kind of files and how much?

Rosetta Project[0] physically etches data into an object for long-term durability. Project used text for human readability as it only requires magnification. One could imagine establishing some OCR/barcode system to enable future re-digitization. Evidently there is now a Nano Rosetta[1] commercial project you can use today to have jewelry printed with your information.

Alternatively, the Arctic Code Vault project[2] evidently used Piql[3] which claims 1000 year durability.

[0] https://rosettaproject.org/disk/concept/ [1] https://nanorosetta.com/ [2] https://archiveprogram.github.com/arctic-vault/ [3] https://www.piql.com/services/long-term-information-storage/

Whether it be this or project Silica, I want my digital permanent storage. Write it in glass, treat it nicely, and yes, you may end up with digital bitrot or whatever, but I want media that stores digital data that cannot physically degrade for hundreds of years, kind of like paper does.

Paperbak being forked and made mainstream, I can accept that. Provided I can also do an automated book printing.

I will also accept holographic stick gum storage a la isolinear chips.

But there has to be more than just tape and blu rays.

Pretty neat, this is worrisome (from the paper): A book was recorded in 4-bit voxels with nearly 100% readout accuracy.

Kind of like "this new car hardly kills anyone." :-) Given that data recovery is like the most significant aspect of any data archival solution.

Not a mention of readout speed either, there are many read-only datasets that could benefit from something like this of course, but if it is way slower than LTO tape I'm not sure exactly where it fits in.

I remember when I first heard about 5D storage back in 2013 [1][2]. The concept isn't at all new; in fact, there's a 5D storage device floating around in a Tesla in space, and Musk has another in his personal library. Nonetheless, it's great to see that it hasn't just dropped off the radar and that developments are still being made. I hope that this technology can one day be as widely available as CDs are; its potential usefulness to archivers/data hoarders really excites me, and (so long as the price is right) who can really sniff at extremely high-capacity, long-lasting storage?

[1] https://www.orc.soton.ac.uk/news/4282

[2] https://doi.org/10.1364%2FCLEO_SI.2013.CTh5D.9

I remember a very similar research project from the 90s. A 3D Holographic cube that was supposed to have a, at the time, insane density.

https://en.wikipedia.org/wiki/Holographic_Data_Storage_Syste...

They do mention parallell writes (multiple lasers at the same time) which would give writing speeds of 96MB/s (500 TB in 60 days).

Which isn't half bad, certainly on pair with standard mechanical hard-drives.

The thpical lifetime of a CD or DVD or BlueRay is about 3-5 years depending on the handling by the user. I'd like to see them outline the claim that this media can last "billions" of years. It sounds highly suspect, as does all the other claims in the article.

It is very durable. It withstood 1000 degree heat for 2 hours in referenced paper, there are no moving parts on the storage itself, it doesn't require additional energy to sustain records, records themselves don't deteriorate over time, and glass by nature is quite resistant to mechanical damage(outer few mm obviously shouldn't be used). If stored correctly, I see no reason for it not lasting a few million years.

Yeah, my first thought even before reading the article, was "What about seek times?". Maybe it can be improved, but thinking of web apps specifically, I'm not sure what kind of use case you would have for something that is this slow to access.

I don't care about seek times if we can have media with orders of magnitude more storage capacity and life expectancy. (of course if you can also have good seek times, it would be icing on the cake)

What kind of data are you trying to store? How much of it? I can’t imagine my life being more than 100 GB and probably much smaller, probably under 20GB.

Scihub, libgen, datasets, etc... as much data as possible.

My personal data is probably less than 1TB at this point, but a way to store it long term would be nice.

This is for archival purposes. Seek times don’t matter for that use case.

Seek times always matter even if they're not the top priority. And especially if we're talking about a storage medium that takes in a whopping 500 TB.

The GP is talking about something interactive like a web app, where a seek time likely would matter. This isn’t intended to be used as such.

Imagine you need to seek 490TB to restore 10TB.

> promising incommensurate storage in a palm-sized device

Incommensurate with what, one might ask.

Leto II approves of this technology :-)

Who comes up with garbage naming techniques such is 5D? What the fuck do they expect people to think when they hear that?

It’s magical storage that goes into the multiverse. Technology is magic!

Reply on news.ycombinator.com