sum[1] = x[1] + y[1]
sum[0] = x[0] + y[0] + carry from previous operation
In contrast, if you'd use 96 bits, you couldn't just use 64 bit integer operations. Instead, you'd have to cast a lot: sum[4..11] = *((int64*) x) + *((int64*) y)
sum[0..3] = (int32) ( (int64) *((int32*) x) + (int64) *((int32*) y) + carry)
So you'd read 32 bit-values into 64 bit registers, set the top 32 bits to zero, perform the addition, and then write out a 32bit value again.It gets much worse if your CPU architecture does not support the addition to 2^(2^n); if you were to use 100 bits, you'd have to AND the values with a bitmask, and write out single bytes.
So 128 is far easier to implement, faster on many CPU architectures, plus you get the peace of mind that your code works for a long time. For instance, let's assume the lower bound of 9 months per doubling (which is unrealistic as described in this article), then you're going to hit:
50 bits (baseline from article): 2004
64 bits: 2014
80 bits: 2026
92 bits: 2035
100 bits: 2040
128 bits: 2062
Now, what's the expected lifetime of a long-term storage system? It's well-known that the US nuclear force uses 8 inch floppy disks. Those were designed around 1970. So a lifetime of roughly 50 years is to be expected. For ZFS, that would be 2054. By this (admittedly very conservative) calculation, 128 bits is only barely more than required.