At which point, it makes sense to just give up and use a normal TLS/QUIC connection. QUIC also has 0-RTT resumption, which is functionally similar to what you want.
Really, raw UDP makes very little sense in today's Internet. It might have been marginally more useful if BCP38/RFC2827 were more widely adopted.
I might agree if the only purpose of UDP was to avoid the handshake. But this issue alone only affects some usecases.
Naive workaround/thought, require the client to pad the first packet to the point where there you can't use it for amplification attacks (not an absurd amount, just 1k or something. Of course depends on the context).
And possibly embed the source IP in the first response so that the indirection isn't as effective either.
The other major use-case for UDP is for protocols where loss is preferable to retransmission delay, it's still very much valid. But in this case, UDP is used within a stateful context, with multi-stage handshakes and everything.
Revocation is indeed a weak point of this solution as it would take time, probably a transaction, to check. This problem might be mitigated by shortening the certificate validity duration.
I don't see why time synchronization would be critical if the validity periods are slightly overlapping.
Seems far simpler to send a physical mail to the service operator who then hardcodes the IP in the server.
Or, maybe do a handshake once and cache it for X amounts of time whatever makes sense for that service.
Edit: If the server creates the certificate with a three way handshake, it will use the remote IP address. So the client doesn't have to know it's IP address