Would Statistical Rethinking help me interpret web app metrics? E.g. if I have a canary out and the response times are longer after x requests, is that significant?
To be clear though, Statistical Rethinking is a beautiful piece of work. You can check out the author's lectures[0] and see how much they suit your needs.
[0] https://www.youtube.com/playlist?list=PLDcUM9US4XdPz-KxHM4XH...
However, for your question specifically, the choice of prior is less meaningful when you have lots of data, and presumably a web app seeing hundreds or thousands of requests per second can gather enough data to determine if the canary has a different latency profile than the deployed version within a few seconds. Also, presumably you would use an uninformed prior for a test like that. If I were trying to prevent latency regressions in an automated deployment pipeline I would just compare latency samples after 1 minute with a t-test or something simple like that.