Scientist: A Ruby library for carefully refactoring critical paths

Maybe because of the large increase in complexity?

They wrote a good blog post about using this gem back in 2016: https://github.blog/2016-02-03-scientist/

There’s goreplay[0]. But once you start using that you quickly find that many parts of your app use random data (such as uuids) or clock data (timestamps etc) getting these synced between prod and replay or ignored on replay is quite a hassle.

[0]: https://goreplay.org/

Another possibility might be to do the side effects in a transaction that gets rolled back?

How would you handle this case WITHOUT this library?

In TDD they might call these pinning tests.

This kind of testing isn’t just good for refactors. You can also use it for rewrites or substantial architectural changes. I think github did something like this while trying to make the storage format more efficient.

Which is really valuable if you don’t have tests, or your test suite doesn’t give you much confidence, is too brittle to make the changes without breaking the tests, or maybe the code is structured so that it’s really hard to test all the paths that occur in production.

Even in a well-tested codebase some integration points are extremely hard to test, notably external services that you connect to. Rewriting these to use a different service or just a different way of connecting to a service would be an example where this ends up being useful.

By the time we extracted Scientist, the code we were refactoring had pretty good test coverage. But even the best test suite is an imperfect model of production. The first section of the README briefly mentions this:

"Let's pretend you're changing the way you handle permissions in a large web app. Tests can help guide your refactoring, but you really want to compare the current and refactored behaviors under load."

See https://github.com/github/scientist/blob/master/README.md#al...