Based on the data, it seems they have 4.4 petabytes of storage under management. Neat.
https://docs.google.com/spreadsheets/d/1E4MS84SbSwWILVPAgeIi...
https://docs.google.com/spreadsheets/d/1E4MS84SbSwWILVPAgeIi...
nbdkit can emulate disks up to 2^63-1 which is also the same maximum size that the Linux kernel currently supports: https://rwmj.wordpress.com/2018/09/05/nbdkit-for-loopback-pt... https://rwmj.wordpress.com/2018/09/06/nbdkit-for-loopback-pt...
Perhaps we don't need a single flat address space with byte-addressable granularity at those sizes?
I wonder how an 8 bit byte, 48 bit word system would have fared. 2*32 is easy to exhaust in routine tasks; 2*48 not so much.
Extended 96-bit pointers could address the (rather exotic) needs of things such as distributed HPC workloads, flat byte addressable petabyte and larger filesystems, etc. Explicitly segmented memory would also (I assume) be nice for things like peripheral DMA, NUMA nodes, and HPC clusters. Interpreters would certainly welcome space for additional pointer tag bits in a fast, natively supported format.
Given the existence of things like RIP-relative addressing and the insane complexity of current MMUs such a scheme seems on its face quite reasonable to me. I don't understand (presumably my own lack of knowledge) why 64-bit was selected. As you point out addresses themselves were 48-bit in practice until quite recently.
History is filled with paging schemes in computers (e.g. https://en.wikipedia.org/wiki/Physical_Address_Extension). Usually people do this initially as it allows one to access more space without requiring a change of all software, it is an extension to an existing software paradigm, but once the CPU can just address it all as a single linear space, it simplifies architectures and is preferred.
My pondering is less "why such a large address space" and more "why such a large native word size"? Extra bits don't come without costs.
As long as I'm asking ridiculous questions, why not 12-bit bytes? I feel like a 12/48 system would be significantly more practical for the vast majority of everyday tasks. Is it just due to inertia at this point or have I missed some fundamental observation?
There are other less publicly well known things with 72 to 96 drive trays in a single 'server' which are manufactured by taiwanese OEMs for large scale operators. The supermicro is just the best visual example I can think of right now with a well laid out marketing webpage.
edit: some photos
https://www.servethehome.com/supermicro-ssg-6047r-e1r72l-72x...
Also some serious cooling to avoid the drives in the front cooking the drives in the back (assuming front-to-back airflow).
You may not, but plenty of people do.
edit: They look like this: https://knowledgebase.45drives.com/wp-content/uploads/2019/0... (image from ddg)
Careful not to drop it.
This is what backblaze use themselves currently able to hold 60 drives .
Not a server per se, but you just take one 1U server and daisy chain a lot of those JBOD chassis for the needed capacity. You can have 1080 disks in a 42" rack.
https://www.backblaze.com/cloud-storage/resources/storage-po...