Going back to David Reed, this is specifically why UDP exists: as the extension interface to build more non-TCP transport protocols.
I am very aware of what you can do with UDP, I have done some very fun work trying to minimize bandwidth usage on crappy mobile connections by using and abusing it. But I think at the end of the day it is an engineering crutch.
If we insisted on properly supporting a diverse set of L4 protocols years ago we wouldn’t have wound up with NAT and slow adoption of IPv6. Address exhaustion would have been a real pressing issue. Instead you can’t even ping a server behind a NAT and firewalls run out of memory trying to manage stateful connections.
UDP is a pretty elegant design for what it is but it is barely good enough to allow us some room to make things work. Ultimately it did limit us more than it enabled us.
And I agree: it stifled what could have been a much nicer to work with set of protocols and who knows what could have been created had we not just said "well there is always UDP if you want to do your own thing".
The fact of ICMP not being itself a UDP protocol caused major problems for systems programmers, because it meant that OS kernels all "owned" ICMP, provided only a baroque sockopt programming interface to like 5% if it, and required userland programs to hold suser privileges to do any real ICMP work. Awful design. And ICMP is slow-pathed by routers, because it isn't UDP.
UDP literally doesn't do anything but multiplex raw IP. Unless you're actually worried about the 8 bytes the header takes up, there's no reason, none at all, to slide a new IP protocol anywhere but on top of UDP. Again: that's why UDP was designed in the first place. You can go look this up! David Reed still talks about it!
we're fighting for bits here!!! if you want to be 100% safe you need 576 packet size, UDP header is 1.4% of that
PMTUD breaks when ICMP is blocked.
The same argument can be made that everything but HTTP being blocked is not a problem because everything can be transported on top of HTTP.
Imagine at the beginning of a connection, sending a burst of packets, you could send (ignoring tcp timestamps because it makes the math hard, PAWS is a waste of bytes for most flows, and etc) [0,1460); [1400, 2800), ...
If you get an ack of the first packet, great. If not, you resend it as a 1400 byte payload and probe again in a future burst. Maybe even premptively resend the first packet as a 1400 byte segment after a short delay. Anyway, have enough failed large packets and probe smaller. Probe bigger again every so often if the connection stays open for a meaningful amount of time.
On top of that, I believe the UDP checksum can be omitted as well at least on some OSes (and is arguably not necessary for fully encrypted/authenticated payloads) – leaving really just the two bytes for the "length field".
If we just allowed all L4 protocol numbers through and ditched NAT we could have nice things. Or we could kick it up two layers to use QUIC for what SCTP could have been.
Having a diversity of IP protocols isn't a nice thing. The designers of TCP/IP made a protocol specifically for doing the thing you wanted to see SCTP do: it was called UDP.