10 petabytes - visualized
blog.backblaze.com
blog.backblaze.com
10 Petabytes is 10,000,000,000,000,000 bytes or 80,000,000,000,000,000 bits, divided by 8,000,000,000 bits per full human genome (2 bits per base-pair) that's about 10,000,000 cells (not red blood cells because they don't contain DNA), or about 5 milliliters! (10 um diameter on average so about 500 cubic um, so 2 million or so per ml), and that includes all the stuff besides the DNA in the cells.
Hard disk storage certainly is impressively compact but it still has a long way to go before we beat mother nature.
-1 already eh? Downmodders please correct my math or say what you think is wrong with the comparison, I note the article ends on: "How would you visualize ten petabytes of storage?".
- Blood is far from being that dense, but I'll assume you mean to calculate a theoretical limit.
- If you use the whole of a cell's DNA to encode information, the cell will die. It can't be used to freely encode information like a hard disk.
- There is no way to retrieve information in this system. This is a bit like packing a large amount of extremely high-density magnetic platters in a box and calling this a storage system, which is very different from using actual hard drives with all the complex reading/writing system (heads, space between the platters, magnets...). Even if they're not connected, it still takes room.
- Reliability of hard disks is far, far better than this. Encoding information as a single copy in a single cell is wildly unsafe. Even using multiple copies, it's likely to degrade with mutations.
Edit: I was under the impression that a base pair could only encode one bit of information, since only T-C and A-G are valid combinations. Wikipedia appears to disagree with me but I see no source; does anyone know why Wiki says a base pair could encode 2 bits?
There are more ways of having cells arranged, I explicitly mentioned red blood cells because they are exceptional in not containing any DNA, but any chunk of tissue with that volume would do.
> If you use the whole of a cell's DNA to encode information, the cell will die. It can't be used to freely encode information like a hard disk.
Yes, that's obvious, but a cell's DNA does hold that much information, it's just not our information.
> - There is no way to retrieve information in this system.
There actually is, the information retrieval mechanism that is used to 'express' the DNA (actually, the RNA, an 'unzipped' strand of DNA, but who's counting) is a wonderful little nano machine called a ribosome, it's probably the most amazing structure that I know of outside of the DNA itself.
They're in the volume quoted, the DNA only occupies about 25% of that volume iirc.
> Reliability of hard disks is far, far better than this.
The error correction mechanism that allows your cells to be copied through very large numbers of generations is actually pretty good, most 'mutations' are lethal and only very few actually result in viable copies passing their changes on to newer generations. Mutations are also pretty rare on the whole.
You are right that only TC and AG are valid, but those combinations can be attached 'in reverse' as well (CT / GA) so that makes for four possible combinations in all.
If it weren't for that the movie 'GATTACA' would have been unpronouncable :)
Actually I was going to say that one way to think of using that information is to compute with it, in other words, an organism is simply the result of a computation on its DNA.
DNA is natively a content addressable storage system, due to natural base-pairing. But to first address your question in your edit: think of DNA as a pair of singly-linked lists, each with an alphabet of four characters. Each singly-linked list is the "reverse complement" of the other: the reverse sequence with an A-T swap and a C-G swap. At each position there's 2 bits of information, and the other linked list allows for some redundancy.
To probe for information in a DNA database, you construct the reverse-complement of the desired bit of information, attach a marker to your probe (such as a fluorescent dye, biotin, magnetic bead), then physically mix it in to your DNA database. A couple cycles of melting and cooling, and your probe will eventually find it's target DNA.
Of course, the thermodynamics of a physical database like this aren't particularly great. I'm not sure of the asymptotic behavior; my intuitive guess is lg(N) just like in B-trees or what have you, but I've never run the numbers or heard of anyone else running it. Also, the constant in front may be just a few orders of magnitude larger than our current systems :)
Reading DNA is getting super cheap these days, and the pace of DNA sequencing technology makes Moore's law look positively wimpy. There are about 30 serious startups working on technologies that fall into a few broad categories, and some like PacBio had an IPO this year. Writing DNA is a much more difficult challenge, I don't know of many people looking into it yet. The market for writing DNA isn't nearly as obvious as it is for sequencing. Of course if it becomes feasible to write your own pets/plants/children instead of breeding them, the market may explode.
To build the reverse-complement of the information, don't you need to have the information in the first place?
I know it's possible to store information in DNA through various means, but I don't believe it can be done at the density the OP calculated. If we're going to take into account only information storage while ignoring retrieval considerations, then we shouldn't compare naked cells with no DNA duplication to reliable whole hard drives.
Call it human pride, but I think we've beaten mother nature in several aspects ;)
And actually, this could be done at a much higher density than what the original poster described, as he's counting the full cell in the density calculation, and DNA is only a small fraction of the cellular volume. You could duplicate all the DNA 10-100 times in the same amount of space once you take out all the ribosomes, proteins and extra water. And as long as it's not stored in direct sunlight or next to your pile of plutonium, DNA is going to be much much more stable than aligning magnetic fields. We're still getting good DNA sequence out of bones that are tens of thousands of years old.
When you think of nanotechnology and miniaturization, think of biology, because that's where all the real nanotechnology is going on. We've not done any better than nature when it comes to making small machinery. Nature has already invented the commodity interchangeable parts (amino acids and nucleic acids) that can self-assemble into rather fantastic machines.
However, we have beaten mother nature on latency: as I alluded to, a DNA database like this would have latency on the order of days for a lookup. On the other hand, as much parallel access as you can imagine is built in, without additional volume. And this isn't a system that has been engineered at all, I'm just talking about the fundamental properties of a little puddle of DNA and water. If half the engineering that went into modern computer hardware were put into a DNA database, it could be quite competitive with our electronic systems.
Well under what nature can do.
You could stack the atoms three dimensionally but then there would be no easy way to read them out again.
-1 already eh? [..]
Backblaze's pods are a data-loss nightmare -- lots of single points of failure which will wipe out many TB of data at a time -- and backblaze has stated that they replicate data across multiple pods. Given that the 10 PB seems to be the amount of raw storage backblaze has, I'm guessing that the amount of actual data stored is much less -- depending on what sort of erasure correction scheme they're using, of course. (They're still much bigger than Tarsnap, of course!)
Agree those SATA port multiplies are worrisome. In the beginning, our prototype machines used them to squeeze as many drives into a single machine as possible. They have unusually low tolerance for electrical interference and make it possible for one badly malfunctioning drive to take an entire array offline until manually serviced. We've seen occasions where just touching a cable attached to a port multiplier caused the Linux kernel to emit "dazed and confused" NMI events. I am not brave enough to try them again, even in a redundant setup.
How did you compute this "replication equivalent"?
Not if they lose 35% on top of that they are around 41% overhead. But, they are taking a huge it on write speeds, network traffic and reliability for doing so.
Edit: Looks like they have 10,058 TB before partitioning the drives so my guess is ~3-6TB of actual user data.
60kW is roughly 80hp. (I know this is a specious comparison in several ways but bear with me.) 80hp is enough power a small but highway-capable motorcycle. A single 60kW brushless DC motor weighs over 100kg. That's heavier than a good many of us HN readers.
Imagine a motor like that bolted to one long spindle of large platters--say the 99cm platters from a 1961 hard drive.
If these platters were remade with the data density of modern 2TB drives, how tall would the spindle be if this giant hard drive had a 10PB capacity? How over- or under-sized would our motor be?
EDIT: And, just a comparison: Google processes over 20PB of data per day!
It's also hard to tell whether they're actively decommissioning their 1TB drives or if they've just on a growth curve and they've done significantly more business while 1.5TB and 2TB drives were the more economical choice, but sooner or later the cost/benefit curves do cross and you do want to start decommissioning the smaller drives, it's only a matter of time.
2280 * 2 + 3166 * 1.5 + 1 * 749 = 10,058 TB.
The skyscraper comparison is fun but misleading. The stack of drives would be very very thin. It would work better if drives were put in one box and that box was compared to a room, house, etc..
There can be other visualizations like electrical power consumption compared to a normal computer. How much time it would take to read/write that data. How many processors are needed to read/write that data in reasonable time, compare their surface area to the our unfolded brain, which is a large cloth napkin.
I think these comparisons would be more useful to people in the field than how many books would be needed to store that much data, the standard pop show example.
These are the questions that keep me up late at night.
http://blogs.sun.com/darren/entry/compress_encrypt_checksum_... - ZFS link
As for de-duping, Dropbox does this. When you drop a Windows ISO or the Office installer into your folder, it hashes the file and realizes 'hey we already have this!' and you don't have to upload it.
10 petabytes : 10000 terabytes :: 10000 gigabytes : 10 terabytes