It makes way more sense to think of an IP address as the entire “address”+port tuple. When you look at it like that, so many aspects of computer networking get simpler. First, we think of every computer on a network as having a range of addresses - usually 65536 of them. But that’s an arbitrary choice that could be adapted in lots of ways. Every connection directly connects two IP addresses - which is way simpler than what we do now. DNS should associate a name with an address - including what we currently call the port. That way we wouldn’t need special application specific port assignments. And then we wouldn’t need SNI and all that - because computers could have a bunch of fully process-isolated web servers, just listening on different incoming address+port tuples.
Obviously we can’t change tcp/ip, but parts of the tcp/ip infrastructure already work this way - but we’re lacking the language to talk about it.