One is quantitative: you have to remember that 2^48 is a much much bigger number than 2^32. With 2^32 IPv4 addresses, you have about 0.5 addresses per human being on the planet, so right away you can tell that stringent allocation policies will be needed. On the other hand, with 2^48 /48 ranges, there are about 8,000 ranges per human being.
So even if you hand a few /48s out free to literally everyone who asks, the vast majority will still be unallocated. A /48 is only about 0.01% of what could be said to be a "fair" allocation. (And yet, a /48 is so huge in absolute terms that even the vast majority of organizations would never need more than one of them, let alone individuals.)
The other is that unlike, say, the crude oil we pump out of the ground, IP address ranges are a renewable resource. If you hand out a free /48 to every person at birth, then long before you start running out of ranges, people will start dying and you can just reclaim the addresses they were using.
It's weird having a subnet size equal to a complete IPv4 Internet worth of IPv6 Internets but I believe the rationale was that you would never in practise run of out IPs in your subnet. A lot of Enterprise IPv4 headaches are managing subnets that are not correctly sized (organic growth, etc.). IPv6 is always routable for the same reason (companies reusing RFC1918 making connecting networks a pain).
There are different headaches with IPv6 - such as re-IPing devices if they move subnet - i.e. move physical location, or during a failover etc.
I'm not sure what the best practise there is as many enterprises don't use IPv6 internally. In my experience anyway.
Incidentally, if you find yourself experiencing an RFC1918 clash, one simple way of fixing it is to use NAT64 to map the remote side's RFC1918 into a /96 from your v6 allocation. You can write the last 32 bits of a v6 address in v4 format, so this leads to addresses like 2001:db8:abc:6401::192.168.0.10 and 2001:db8:abc:6402::192.168.0.10, which don't overlap from your perspective.
(If you wanted something simpler to type you could put them at e.g. fd01::192.168.0.10... but then you do start running the risk of collisions with other people who also thought they could just use a simple ULA prefix.)
Really, just adding 2 bytes to IPv4 would have fixed everything and made it a lot simpler to move over. IPv6 is overkill and I think that really hurt its adoption. I remember being at uni and being told "this is the next big thing". In 1993. And it's not even a big thing now. Not on the user side anyway, I can still access everything from IPv4.
Considering how hard deploying a new L3 protocol is, we're only going to get one shot at it so it's a lot better to end up with too many addresses rather than too few.
L3 has nothing to do with MAC addresses anyway so I've always found that a pretty weird decision anyway. Sure, it avoids having to implement ARP but we need that again now anyway with the randomisation. And ARP is like a one-time in a few minutes kinda thing anyway.
I'm pretty sure that if we'd just gone for "a couple bytes extra" we'd have long been completely over. It's the whole L3 transition itself that suffers from the complexity. I remember it well in the 2000s, nobody in telecoms wanted to touch it. And when IPv6 was invented in '93 or so, the installed base was extremely small. It'd have been a piece of cake to get it over with then.
The aggregation necessarily "wastes" L3 addresses, so if you think you'll have enough machines to justify an L2 address size of n bits then that also implies needing an L3 address size of n+m bits, where m is a number that represents how densely packed your L3 address space is. Anything smaller than that will be too small to handle the full extent of your L2 address space.
RFC 3194 (https://www.rfc-editor.org/rfc/rfc3194.html) suggests we'd want something like 80 bits to handle EUI-64. 128 bits is the next power of 2 up from there.
> It was a pretty arbitrary decision to accommodate the MAC address inside the IPv6 address [...] Sure, it avoids having to implement ARP
You're thinking of SLAAC, which picks the address by slapping the MAC/EUI-64 into the right-hand 64 bits, but this is just a convenient way of picking the address bits. There's no special significance to those bits and you still need to do ARP.
> I'm pretty sure that if we'd just gone for "a couple bytes extra" we'd have long been completely over.
We still can't get people to stop hardcoding socket(AF_INET, ...) or manually crafting sockaddr_in structures. This is the minimum amount of work that will always be needed, regardless of how many extra bits are involved, and even this part hasn't been quick.
Either, NATv6 would become a thing, or instead I hope SLAAC would get deprecated and dhcpv6 would become mandatory so we could give out smaller than /64s