Scientist: A Ruby library for carefully refactoring critical paths

We've used this extensively at my work and it has been great. As someone else pointed out it's a bit tricky if the code isn't idempotent. We used this as part of a data migration project and for writing we ran the experiment 100% of the time and read back the result to check it was correct - a bit of a performance impact but nothing that was noticeable. One super useful extension we've made to the gem is to have it update Prometheus metrics so you can easily see how often your experiment is showing that the candidate is different from the control.

The readme (here https://github.com/github/scientist#alternatives) doesn't mention, but here is one for Rust: https://crates.io/crates/scientisto

I’ve used this gem to remove a Rails Model that was heavily tangled and used in production. It had a bit of a learning curve, such as seeing how errors get handled in try {} blocks, but really nice otherwise when doing the equivalent of open heart surgery.

This is pretty cool, but in my experience critical path code generally has side effects and most code is not idempotent. That is to say it may not be safe or feasible to run the old and new code in parallel, without some special considerations. Think bidding on a auction item or transferring a bank balance.

Can anyone share their experience of the equivalent in Go? https://github.com/technoweenie/go-scientist

This is fantastic! Do folks do this manually in other languages or are there proven packages folks use?

previously: https://news.ycombinator.com/item?id=21489877 (178 pt, 19 comments)

Dear lazyweb, is there a similar library for python? :)

Pretty cool to see the github of Github, never occurred to me but makes total sense.