ZFS 128 bit storage: Are you high? (2004)
blogs.oracle.com
blogs.oracle.com
So exactly 14 years passed, does someone have 2^64 bytes for a single ZFS filesystem (or anything close to that)? I don't really feel like storage capacity (or 1/price) doubles every year.
So with 5 doublings, if we saw datasets on the order of 2^50 back then, we should see datasets on the order of 2^55 today (30 PB). And sure enough, here is a computer with a 30 PB global filesystem: https://www.fujitsu.com/downloads/TC/sc11/k-computer-system-...
So another 9 doublings to go to 2^64 bytes. With one every 34 months, this should happen in 25 years (2043). But again, if you are a filesystem developer you should be on the conservative side and assume it might double every ~12 months.
If ZFS lives as long as the FATs have (three decades) even at a doubling every 34 months, that means six remaining doublings before 2035. Not quite 2^64, but that's on the liberal side. Half that (doubling every 17 months) is twelve doublings, which is easily over 2^64.
;)
That has to be one of the worst typos I have ever seen.
Of course, the real reason for the mistake is that phone spelling correction will convert endian to Indian.
That’s 7 PB to 19.2 PB. If they had replaced all 3,500 drives it would be 4X.
I think that's the point from some of the other comments (below). Instead of having every larger and larger single filesystems, once you get to a certain point (~1-2PB is my guess), it simply isn't worth it to organize your data under a single filesystem.
That's not to say ZFS was wrong in choosing 128 bits... at the time ZFS was designed, the level of horizontal partitioning that we are now accustomed to just wasn't a thing. Compared to HDD times, networks were far too slow to support such a thing. Now, however, we talk about HDDs being too slow compared to our networks. At the time, 128 bits was a solid choice, even if it was overkill.
Plus, Sun was definitely a scale up kind of company... if you want to get bigger, by a bigger ($$$$) box. So having a large capacity FS was good for their bottom line. Now, we think in terms of scaling out with multiple (cheaper) smaller boxes... which is probably the only way we've managed with current levels of data.
No, that is not true. A vault acts as a single unit, effectively making it a single filesystem as much as a ZFS pool is a single filesystem.
Internally, a vault operates on a set of "tomes", which are groups of 1 harddrive from each storage pod (20). These "tomes" handle the redundancy aspect as well. However, the "tome" is not the filesystem.
The concepts map cleanly to the ZFS world: A vault is equivalent to a ZFS pool, and a tome is equivalent to a ZFS vdev in redundant configuration. Neither vdevs nor tomes are filesystems on their own—they only become a filesystem when combined into pools and vaults, respectively.
Now, your argument does hold if interpreted differently: One does not use a normal filesystem for those absurd sizes. However, it is a single filesystem.
I guess I was assuming it worked like Lustre or BeeGFS where there were smaller FSs (ext4, XFS) at play that handled storing file (chunks) and then a higher level interface that managed it all.
Either way -- the original comment was referring to Backblaze swapping 7PBs for 19PBs. Which we agree is not a normal size for a "normal" file system! ZFS is a strange beast... it was designed to support a world that just never showed up. It is a "normal" FS that works from laptops to servers, but was designed to support a big iron world that never really appeared (much). Instead, here we are comparing it not to XFS, but to cloud FS's that were designed for a completely different "cloud" world.
I would assume that the tome, beneath network protocols, interacts directly with the individual block devices. They have implemented things like Reed-Solomon error correction themselves to handle redundancy across the tome (which to recap is their cross-server vdev equivalent), so they have indeed implemented tasks of a filesystem and RAID manager on top of their lower-level primitives.
I also believe that what their low-level primitives are is actually a rather irrelevant implementation detail. A filesystem presents a way to store and organize files, and is a filesystem regardless of whether it is implemented on a block device or an egg-engraver. It could be using FAT16 as object store, sharding across files and handling metadata elsewhere. I would still consider such setup to be a true filesystem, as it is not simply a 1:1 network-to-disk protocol.
And yes, ZFS won't see the large systems it is designed to handle anytime soon. What Sun did not see coming was the death of large, high-capacity servers, and the rise of distributed solutions. But then again, no one saw that coming in 2001 when ZFS development started.
However, considering that storage sizes will continue to increase, who knows what will happen in the future... 128 bit filesystems might come in handy at some point.
(Note that my information on backblaze's systems is based on public information—I am not an insider of any sort.)
Is that a fundamental thing or is it just because there aren't many filesystems that can handle it? It seems to me like a single filesystem would be conceptually easier, but I'm far from knowledgeable in this area.
But really, I do think it's physics at play... or rather logistics.
Let's say you're trying to put together a 10PB storage system in a datacenter. If you're using 60 disk JBODs with 10TB drives, thats 600TB of raw space per 4U (or a max of 6PB per rack, or 12PB in 2 racks). Now realistically due to power, network gear, etc... instead of maxing out 2 racks, you'd probably split that into at least 4 (or more) racks. If you're Backblaze (see above), you split those 20 pods across 20 separate racks (and add more volumes, but that's another story).
But once you're at that level, for redundancy, you'd not only managing which disk data is written/replicated to, but also the JBOD/pod, and the rack (and the data center). So, at that point, you're not going to working with a "normal"/traditional file system. You could with ZFS define redundancies that would work like that, but realistically, because of the extra overhead, you're better off changing how you organize your data.
(Plus: ZFS isn't very good at expanding a filesystem with new storage. You can do it, but your pool ends up unbalanced and performance suffers. When you decide to have a 1+PB system, you normally want to have the ability to add more storage space, which means a different type of FS)
So basically we moved away from singular large unified file systems and built swarms of little file systems.
I can't certainly blame their choice though, and 16-million terabyte hard drive arrays (the limit of 64-bit addressing) on a large mainframe are really just a few quantum leaps away. Heck, I still vividly recall buying a MASSIVE 40 GB HDD in 2000 - it seemed excessive beyond wildest dreams at the time (and was promptly filled to brim at a LAN party.)
Since, advances in mass storage tech, most importantly the successful commercialization of perpendicular recording in 2005, mean that nowadays 12 TB drives are commercially viable - roughly 1000x the size of a 40 GB drive.
In the next 20 years, petabyte-sized drives might well be available (though almost certainly not based on any form of magnetic recording). Once you hook hundred 50-petabyte drives to a single fileserver, you start closing in on limitations of 64-bit addressing.
Everybody uses it to mean "big". Oh, language.
The point is, that the increase in frequency is so large, that using MHz is no longer sensible.
There has been a "quantum leap" in frequency.
I assumed it implied something far 'larger' such as a leap through spacetime...
It typically implies a revolutionary change over evolutionary improvement.
The interesting thing there was that the electrons existed only at discrete energy levels, and the transition between them appeared instantaneous.
No cloud vendor has a billion harddisks, that's for sure.
(The filesystem might only support a maximum file size of 2^64 bytes, but that's nothing to do with the capacity of the storage array)
ZFS aside, the 512 byte sector isn't even a hard limit at the storage medium level. Disks with 528 byte sectors — and other odd sizes — have been available in the enterprise world for a long time now: https://en.wikipedia.org/wiki/Hard_disk_drive#Market_segment...
HDDs and SSDs with 4 kB sectors dominate the consumer market now: https://en.wikipedia.org/wiki/Advanced_Format
When you're looking at the same price for two systems, and one is 100x-1000x faster at the cost of being 'merely' hundreds of TB per gallon instead of a few PB per gallon, nobody is going to bother engineering the latter just to save a few bucks on shipping.
https://cloud.google.com/files/storage_architecture_and_chal...
No, they're not a single filesystem, as the papers make clear here and there.
Rates were quicker back around 2000-2005, but that pace has fallen dramatically.
https://en.wikipedia.org/wiki/Mark_Kryder#Kryder's_law_proje...
May I remind you that flash, DRAM and SRAM already are 3D structures.
I calculated it at some point and it was several years of the entire earths manufacturing capacity for disks in a single filesystem.
[added the missed "several years of"]
Or, exactly 2 Zettabytes.
—Wikipedia
ZFS being an acronym for Zettabyte File System was neither retrofitted nor all that brief in use it seems.
Former employees of Sun Microsystems, I know some of you browse HN. Care to shed some light on this?
There is also EOS[2] which is based on XRootD and is planned to replace AFS[3] as a general purpose networked filesystem.
[1] http://xrootd.org/ [2] http://eos.web.cern.ch/ [3] http://information-technology.web.cern.ch/services/afs-servi...
Moore's law only ever really held for transistor density on integrated circuits, not magnetic storage. Magnetic storage capacity always grew more slowly.
Flash storage capacity today has more to do with die stacking than transistor density.
In any case, Moore's Law is definitely dead now. Like, for real this time.
https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/
They also link to this great (logarithmic) chart:
http://www.mkomo.com/cost-per-gigabyte-update
TL;DR: price hasn't been dropping by half annually, but prices have come down from ~$0.70/GB to < $0.03/GB now. So roughly a factor of 20 over 13-14 years, somewhere in the neighborhood of 25% improvements compounded annually over that time.
Its unlikely. There are few opportunities to compile a data set of that size. One might be the NSA collections center in Utah, one might be Google or Microsoft's web cache, and one might be something like the Internet Archive's cache.
However the length choice gets 'weird' if you want something larger than 64 bits and less than 128. Your next intermediate choice is 96 bits. That is 64 + 32 (or one additional 32 bit long word. Growing by just 8 means you now have a 9 byte pointer, by 16 gives you a 10 byte pointer. Some architectures are penalized when doing off word alignment, and so on those architectures the structure is padded out to the next word size anyway.
I've always found Jef's reasoning in that post amusing but I continue to thing 64 bits would have been fine (just like I think it would have been fine for the Internet v6 work :-). Fortunately time has given us a crap ton of memory for our boxes so the storage penalty isn't too great.
The on-disk format could even remain the same; it wouldn't really matter except for lots of small files and those are already so horrid that ZFS might have some other way of handling them. (Like treating /all/ small file names and data as part of a larger directory file or something.)
Even if we ignore that factor of 800 or so (400PB spread across 500 nodes), they still would need another factor of 128,000 before they would need ZFS's 65th bit.
Keep in mind it's 2^128 blocks, not 2^128 bytes.
Backblaze would need to consume the entire planets supply of disks for a very long time AND put all those disks connected to a single linux, freebsd, or solaris box. Only then would they need the 65th bit that ZFS has for addressing blocks.
https://www.anandtech.com/show/10098/market-views-2015-hard-...
I don't know about that exact disc/format, but if we and as we are finally able to stably write at such capacities... (whatever technologies end up being used to do so).
We're going to run into another "640KB should be enough for anyone", unless we are forward thinking with regard to potential storage format capacities.
(And if you think no one will ever be able to consume that much data and detail, just think of all the modeling that will be done and the data behind that. Think of the data being put out by CERN and not just how it will expand but also how people will want to explore lower significance hints, corner cases, and who knows what. Etc.)
https://www.emc.com/collateral/hardware/white-papers/h10719-...
So why worry? We’re doomed to rewrite it anyway :)
OTOH it guarantees there wont really need to be on disk changes for pointer sizes. That may be useful in situations that weren't really imagined, for instance 128b is enough for a Single Level Store where persistent and working set (or NV) are all in the same mappable space for any conceivable workload.
Having twice as much space occupied has many implications: twice as much space lost for data itself, less metadata will fit CPU caches, less throughput in buses between CPU and storage, lengthier read time for the storage itself, etc.
So they used 128 bit because of bikeshedding. Committees always make the most conservative decision possible. Like IPv6.
Creating unified namespaces is really useful and a _great_ simplifier. The reason we don't do that as often as we should is because of limitations in various layers of modern software stacks, especially in the OS layers.
Unfortunately, AFAIU ZFS only supports 64-bit inodes. A large inode space, like 128-bit or even 256-bit, would be ideal for distributed systems.
Larger spaces for unique values are useful for more than just enumerating objects. IPv6 uses 128 bits not because anybody ever expected 2^128-1 devices attached to the network, but because a larger namespace means you can segment it easier. Routing tables are smaller with IPv6 because its easier to create subnets with contiguous addressing without practical constraints on the size of the subnet. Similarly, it's easier to create subnets of subnets (think Kubernetes clusters) with a very simple segmenting scheme and minimal centralized planning and control.
Similarly, content-addressable storage requires types much larger than 128 bits (e.g. 160 bits for Plan 9 Fossil using SHA-1). Not because you ever expect more than 2^128-1 objects, but because generating unique identifiers in a non-centralized manner is much easier. This is why almost everybody, knowingly or unknowingly, only generates version 4 UUIDs (usually improperly because they randomly generate all 128 bits rather than preserving the structure of the internal 6 bits as required by the standard).
ZFS failed not by supporting a 128-bit type for describing sizes, but by only supporting a 64-bit type for inodes. And probably they did this because 1) changing the size of an inode would have been much more painful for the Solaris kernel and userland given Solaris' strong backward compatibility guarantees, and 2) because they were focusing on the future of attached storage through the lens of contemporary technologies like SCSI, not on distributed systems more generally.
HDFS, QFS,.... even old GFS
You wouldn’t make them Linux/fuse mountpoints though, that’s just an unneeded abstraction. Command line tools don’t work with files that are 100TB each.
Command line tools don’t work with files that are 100TB each.
100TB files are often databases in their own right, with many internal objects. But because we can't easily create a giant unified namespace that crosses these architectural boundaries, we can't abstract away those architectural boundaries like we should be doing and would be doing if it were easier to do so.
A real world example are pointers on IBM mainframes. They're 128 bits. Not because there's a real 128-bit address space, but because the pointer also encodes information about the object, information used and enforced by both software and hardware to ensure safety and security. Importantly, this is language agnostic. An implementation of C in such an environment is very straight forward; you get object capability built directly into the language without having to even modify the semantics of the language or expose an API.
Language implementations like Swift, LuaJIT, and various JavaScript implementations also make use of unused bits in 64-bit pointers for tagging data. This is either not possible on 32-bit implementations, or in those environments they use 64-bit doubles instead of pointers. In any event, my point is that larger address spaces can actually make it much easier to optimize performance because it's much simpler to encode metadata in a single, structured, shared identifier than to craft a system that relies on a broker to query metadata. Obviously you can't encode all metadata, but it's really nice to be able to encode some of the most important metadata, like type.
Moreover the convention that there should be no subnet smaller than /64 enables stateless auto configuration for hosts. 64 bits is enough to fit common (supposed to be unique) hardware identifiers and even is large enough to assign random addresses (like with privacy extensions) with a very low probability of collisions.
IMHO, the 128 space was a big mistake. It's twice as hard to communicate between humans, most languages and databases don't support the data type natively and it complicates high-speed routing.
An average of 48 bits for a network and 15 for the host would have been better. For other reasons you almost never want more than a few hundred hosts on one layer-2 network anyway.
Compare to a 3 bit network. By moving IPs from hub to hub, all five devices can always get an address, but the small hubs need to communicate which addresses they own to the outlink and to each other in order to avoid address exhaustion on either hub. Routing a packet is slower because the routing is more complex.
How much more efficient is the 128 vs 64 for routing and what trade offs does it make for other things? I’m now wondering.
This way we have ~1 IP address per 6 micrograms in the solar system, or per 3.4 tonnes in the galaxy.
This is actually an argument against 128 bit, because it clearly shows that 128 bit are unreachable and thus a waste. What about 96 bit?
The real risks of economic damage caused by bit-fiddling bugs is much greater than the risk of bringing the universe’s thermodynamic heat-death ever so slightly nearer...
Can someone explain this? Is there some kind of awkwardness/waste with anything less than doubling the number of bits?
sum[1] = x[1] + y[1]
sum[0] = x[0] + y[0] + carry from previous operation
sum[4..11] = *((int64*) x) + *((int64*) y)
sum[0..3] = (int32) ( (int64) *((int32*) x) + (int64) *((int32*) y) + carry)
It gets much worse if your CPU architecture does not support the addition to 2^(2^n); if you were to use 100 bits, you'd have to AND the values with a bitmask, and write out single bytes.
So 128 is far easier to implement, faster on many CPU architectures, plus you get the peace of mind that your code works for a long time. For instance, let's assume the lower bound of 9 months per doubling (which is unrealistic as described in this article), then you're going to hit:
50 bits (baseline from article): 2004
64 bits: 2014
80 bits: 2026
92 bits: 2035
100 bits: 2040
128 bits: 2062
For instance, consider this C code for adding two 96-bit numbers on a 64-bit machine (ignoring carry for now):
#include <stdint.h>
extern void mark(void);
int sum(uint64_t * a, uint64_t * b, uint64_t * c)
{
mark();
*c++ = *a++ + *b++;
mark();
*(uint32_t *)c = *(uint32_t *)a + *(uint32_t *)b;
mark();
return 17;
}
callq _mark
movq (%rbx), %rax
addq (%r15), %rax
movq %rax, (%r14)
callq _mark
movl 8(%rbx), %eax
addl 8(%r15), %eax
movl %eax, 8(%r14)
callq _mark
Taking out the (uint32_t * ) casts to turn the C code from 96-bit adding into 128-bit adding generates assembly code that only differs in that both movl instruction become movq instructions, and addl becomes addq.
So, if you were writing in C it looks like a 96-bit add would be a little uglier than a 128-bit add because of the casts but isn't slower or bigger under the hood. But note that this is assuming accessing the 96-bit number as an array of variable sized parts. It's that assumption that introduces the need for ugly casts.
If a struct is used, then there is no need for casts:
#include <stdint.h>
typedef struct {
uint64_t low;
uint32_t high;
} addr;
extern void mark(void);
int sum(addr * a, addr * b, addr * c)
{
mark();
c->low = a->low + b->low;
mark();
c->high = a->high + b->high;
mark();
return 17;
}
(I still have no idea how to handle the carry in C, or at least no idea that is not ridiculously inefficient. When I've implemented big integer libraries I've either used a type for my "digits" that is smaller than the native integer size so that I could detect a carry by a simple AND, or I've handled low level addition in assembly).
uint64_t n = 0xdeadbeef;
uint32_t foo = (uint32_t)n; // OK
uint32_t *bar = (uint32_t*)&n; // "OK" but useless
foo = *bar; // undefined behavior!!!
uint8_t *baz = (uint8_t*)&n;
uint8_t byte = *baz; // OK, uint8_t is `unsigned char`
// Same-size integral types are OK
const volatile long long p = (const volatile long long*)&n;
const volatile long long cvll = *p; // well-defined
Don't most C compilers support a pragma to control this? "#pragma pack(4)" for clang and gcc, I believe.
Given this (where I've made it add two arrays of 96-bit integers to make it easier to figure out the sizes in the assemply):
#include <stdint.h>
#pragma pack(4)
struct block_addr {
uint64_t low;
uint32_t high;
};
int sum(struct block_addr * a, struct block_addr * b, struct block_addr * c)
{
for (int i = 0; i < 8; ++i)
{
c->low = a->low + b->low;
c++->high = a++->high + b++->high;
}
return 17;
}
movq (%rbx), %rax
addq (%r15), %rax
movq %rax, (%r14)
movl 8(%rbx), %eax
addl 8(%r15), %eax
movl %eax, 8(%r14)
movq 12(%rbx), %rax
addq 12(%r15), %rax
movq %rax, 12(%r14)
movl 20(%rbx), %eax
addl 20(%r15), %eax
movl %eax, 20(%r14)
movq 24(%rbx), %rax
addq 24(%r15), %rax
movq %rax, 24(%r14)
movl 32(%rbx), %eax
addl 32(%r15), %eax
movl %eax, 32(%r14)
...
movq 84(%rbx), %rax
addq 84(%r15), %rax
movq %rax, 84(%r14)
movl 92(%rbx), %eax
addl 92(%r15), %eax
movl %eax, 92(%r14)
Changing the loop to 4 iterations for compactness' sake, (aligned) structs of two u64s generate the following, vectorized code:
vmovdqu (%rsi), %xmm0
vpaddq (%rdi), %xmm0, %xmm0
vmovdqu %xmm0, (%rdx)
vmovdqu 16(%rsi), %xmm0
vpaddq 16(%rdi), %xmm0, %xmm0
vmovdqu %xmm0, 16(%rdx)
vmovdqu 32(%rsi), %xmm0
vpaddq 32(%rdi), %xmm0, %xmm0
vmovdqu %xmm0, 32(%rdx)
vmovdqu 48(%rsi), %xmm0
vpaddq 48(%rdi), %xmm0, %xmm0
vmovdqu %xmm0, 48(%rdx)
retq
vmovdqu64 (%rsi), %zmm0
vpaddq (%rdi), %zmm0, %zmm0
vmovdqu64 %zmm0, (%rdx)
vzeroupper
retq
movq (%rsi), %rax
addq (%rdi), %rax
movq %rax, (%rdx)
movl 8(%rsi), %eax
addl 8(%rdi), %eax
movl %eax, 8(%rdx)
movq 16(%rsi), %rax
addq 16(%rdi), %rax
movq %rax, 16(%rdx)
movl 24(%rsi), %eax
addl 24(%rdi), %eax
movl %eax, 24(%rdx)
movq 32(%rsi), %rax
addq 32(%rdi), %rax
movq %rax, 32(%rdx)
movl 40(%rsi), %eax
addl 40(%rdi), %eax
movl %eax, 40(%rdx)
movq 48(%rsi), %rax
addq 48(%rdi), %rax
movq %rax, 48(%rdx)
movl 56(%rsi), %eax
addl 56(%rdi), %eax
movl %eax, 56(%rdx)
retq
IIRC, unaligned loads and stores will also, at the hardware level, stall the pipeline and inhibit out-of-order execution.
New link: https://godbolt.org/g/8uGn4h
I believe this program properly handles carry from the low to high part.
The 96- and 128-bit code have the same number of instructions, but the 128-bit code has more instruction bytes due to "REX prefixes" (i.e., 32-bit register add is 3 bytes of opcode, 64-bit register add is 4)
In real-world use memory needs will vary by use like anything else, but are entirely reasonable. I have an old box with 4 gigs of RAM and about 20T disk that performs just fine for modest file-serving needs. If I had more than a few users for that system, it would need more at some point, but mostly for client access software, not the filesystem.
ZFS will use as much memory as you want, and benefits from lots of it in many use cases. But it doesn't require it.
I find it to be the current sweet-spot between useful features, performance, and stability - snapshots, trivial filesystem serialization; not the fastest, but acceptable; and rock solid.
Anyone approaching it from scratch, I highly recommend thoroughly going through the operations one does without your irreplaceable data on it. Everyone, including me, ignores this recommendation. So at the very least, allow me to suggest that you think very long and hard before running that -f (force) command the first time you replace a disk on the array with the baby pics.
[1] Maybe on a system with a lot of ram but constrained storage; I haven't encountered those, but I'm sure they're out there somewhere.
The high memory usage profile definitely isn't from an extra 4 bytes on a pointer, but from design and features of the filesystem.
So 2^64 * block size = 2417851639229258349412352 bytes
Or 2147483648 PB. Sure there might be distributed systems that are mind boggling large, but those aren't in a single filesystem attached to a single node.
ZFS would be a better filesystem with "only" 64 bit blocks.
Today's higher density nodes are something like 48 disks * 12TB, so approximately 2 per PB. 12 per rack would be 6PB. So 178,956,970 racks consuming 300 times the annual production on earth and you'd start wishing for that 65th bit. All in a single OS connecting to a single storage system.
Or you could assume that in the next few decades you'd be batshit crazy to want to install that much storage on a single zfs system and half the overhead of every pointer into the filesystem.
2 to the n...
0 = 1
1 = 2
2 = 4
3 = 8
4 = 16
5 = 32
6 = 64
7 = 128
A decade later, we have a lot of decent compression schemes out of patent and, likely based on their now public methods, a revitalization of development towards schemes that are more optimal for different use cases. Some combination of pre-filtering stages and compression for large caches of small files (like a directory of source code objects or configurations that are infrequently read) might make sense today.
So we should be good until OceanCoin is introduced.
We still have north of 7PB of ZFS-based storage at $DAYJOB (NexentaStor based and Dell hardware) after introducing it nearly eight years ago. ZFS itself has been awesome.
...really?