The OP states: "When it’s time to send an update to the state, XOR it against the previous state, .. and send it." and this is not going to work if the previous state just isn't there.
It would be easy to fix: add a base-version number to the update. If the client doesn't know about the base-version it knows it must not update its state, but rather fetch a full state again or something.
Your suggestion "make sure there is a safe base version available and then send a diff against that" makes a lot more sense.