It's not just any memory. When it comes to core infrastructure routers those routes need to fit into specialized and expensive CAM (Content Addressable Memory) to do the lookups in hardware. And on every single one.
On the IPv6 side; by 2002, nobody was really experimenting with A6 records any more, and EUI64 was needless. Both were parts of IPv6 designed to facilitate "easy" renumbering, so that single prefixes could be replaced with larger ones. But the ISPs weren't complaining any more about table size.
Routers had to get better (more tcam capacity) because there wasn't much choice. Nobody wants to run two border routers each with the table for half the /8s or something terrible like that. And you really can't aggregate /24 announcements when consecutive addresses are unrelated.