It is kinda fun to work directly at IP level and not use TCP or UDP at all. I highly recommend giving it a try.
Possibly dumb question: if you're not using ports (because you're not using TCP at all), what does routing look like, in terms of this article? How does the kernel decide which traffic goes to application A and which traffic goes to application B? Or is all traffic directly visible to each application, if they aren't bound to a port? Can you bind to a specific IP, or just to an interface?
Not sure of the implementation details, but in principle the same logic about specificity of socket binding could be done by the kernel when deciding which socket to actually send the data to. Any other logic (CRC validation, packet ordering, confirming packets have been received, MSS etc.) are all disabled, you just get the raw packet as it was received on the wire, based on the IP headers (an the same happens when you send a packet on a raw socket).
Most of that address space is used to allow for stateless autoconfig, though. It would be certainly possible to fit in port multiplexing, but you'd have to make a trade-off in increasing collision risk, depending on how you do it. Or go back to static addresses or DHCPv6.