How do you deal with the fact that the patch operations are not associative? This seems like a recipe for conflicting effects... you don't want the result to depend on the order you picked up items. Do you batch the effects? (e.g. deletes, then inserts, then modifications?)
What is the behavior of your chosen library--mutable or immutable operations? I.e. do you pay the penalty of making a deeply nested copy of your scene graph for each turn?
Asking for a friend....