It's a fun question, but I think it has a straightforward answer.
1. The IPv6 header is already twice as big as the IPv4 header, so the win in shrinking the TCP or UDP headers is extremely marginal.
2. No, the kernel IP handling code would get more complicated; since BSD Unix, the PCB code in the kernel has assumed sessions were organized by port numbers. Meanwhile: the underlying concept of a numeric mapping between applications and instances of protocols on the wire wouldn't go away; you'd just be overloading IP addresses to accomplish it.
3. I agree that the opportunity to encode funny service names in hexadecimal trailing address space is an upside to eliminating ports.
4. Removing ports strictly simplifies network reconnaissance. IPv6 already makes scanning painful, whether or not you have TCP/UDP ports. With IPv6 and TCP ports, you have to do both: identify the live hosts, and then identify the listening ports on that host.
Some architectural downsides:
1. It is often (not always) helpful to distinguish between an entity providing a service and the services they provide. That distinction is lost when everything is an address.
2. Port numbers prevent us from accidentally trying to speak, say, RTP to a line-printer protocol server. Badly coded servers will often crash given random crap, and while that's not dispositive (the priority should be "eliminate badly coded servers), it's a helpful feature of having ports.
3. The network is required to be aware of which end-systems own which addresses. They are not required to be aware of which applications are running on which end-systems. The end-to-end argument says: put the minimum amount of state in the network, leave as much up to the end system as possible.