It's already typical to have multiple IPv6 addresses per interface, due to privacy extensions. You can see which IP address an application is listening on with netstat in the same way you see the port number. You would need to allocate addresses differently—each host would get a /112 instead of a /128 so that the port number can be placed in the lowest 16 bits, and the tooling (e.g. "ip addr") would by default show the 112-bit prefix assigned to the host rather than hundreds of individual /128 addresses.
The real casualty from merging IP addresses and ports, however, is that it would no longer make sense to listen on a certain port on "any" address/prefix (::/0); if there is more than one interface or prefix assigned to a host and an application wants to accept incoming connections from each of them it will need to listen on multiple sockets, and open and close those sockets as prefixes are added or removed.