At which point, it makes sense to just give up and use a normal TLS/QUIC connection. QUIC also has 0-RTT resumption, which is functionally similar to what you want.
Really, raw UDP makes very little sense in today's Internet. It might have been marginally more useful if BCP38/RFC2827 were more widely adopted.