Adding latency: one step, two step, oops
blog.lawrencejones.dev
blog.lawrencejones.dev
In this case, finding a way to experiment in production was a great way of derisking the work we needed to do.
Even so, we didn't jump at it straight away: there's a lot you can do with data alone, and even when you're really cautious, production doesn't come without risk.
Some strategies like instrumenting the database queries are valuable in a number of situations, and I hope this might help someone in a similar situation.