just XOR-ing doesn't cut it. The client might receive the state out of order, some updates might be missing completely - all of this must be handled somehow
It would be easy to fix: add a base-version number to the update. If the client doesn't know about the base-version it knows it must not update its state, but rather fetch a full state again or something.
Your suggestion "make sure there is a safe base version available and then send a diff against that" makes a lot more sense.
Maybe worth noting that if your world has some known starting state (eg after map load), it might be worth the client and server using that as the implicit initial state, even when a client joins a game in progress.
And after this, you're already half-way to TCP. Why not just use TCP for these updates anyway instead of running your own implementation?