Feeding production data into two different paths to compare them is obviously valuable. Does anyone have any good reasons why this has to happen in online code and can't happen against database replicas in offline mode?

I always feel that it would be great to be able to do this with code that has side-effects (eg. anything that changes the database) but I've never seen a general purpose solution for this. The README mentions using a write replica, but how do you deal with data drifting in case of bad writes?