Hacker News Reader

> If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way.

You seem to be assuming 1. some sort of large-horizontal-scale production system with multiple customers, where the impact of a failure can be minimized by minimizing the number of users exposed to new features, and where 2. there's no type of bug in the code that would potentially take down production as a whole.

What if your production system is, say, a bank's ACH reconciliation logic? A medical device? A car? The live server for a popular MMORPG? A telephone backbone switch? A television or radio broadcast station?

In these cases, your software isn't a service with multiple distinct customers that each make requests to it, where you can test your new code on one customer in a thousand; your software is just running and doing something—one, unified something per instance of the system (though that process may track multiple customers)—and if the code is wrong, then the whole system the software operates will fail.

How do you test software for such systems?

Usually by having a "production simulation" whose failure won't kill people or cost a million dollars in lost revenue.

Thank you for contrasting life building the latest social media platform from what many of the rest of us do.

Currently I work on systems to prepare and validate birth and death certificates for the state, counties, hospitals, et al, and this whole “throw it against the wall and see what sticks” methodology doesn’t fly. Nor would it have worked when preparing and presenting investment account information 5 years ago, nor the job 10 years ago processing lawsuit and insurance claim cases and legal bills. Nor any place that I at least have ever worked in the last 30 years.

Agree, and I work in the latest 'social media platform' type end. We have many customers. I can assure the author of the post, when those customers pay for enterprise licensing and their system is broken with an obvious bug. The 'we didn't do any testing before hand because staging is a lie' doesn't actually fair well as a valid excuse for anything. In fact, you just look like an unprepared and immature muppet.

Well in that case you're not talking about the original scenario where we were talking about whether or not to use a pre-production setup that mimicked production as closely as possible, are you?

> If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way.

Basically you're outsourcing QA to your customer. Some systems may afford this, others not.

> Basically you're outsourcing QA to your customer.

You've just described any software ever used.

Not in any meaningful way is that true, no.

>> You need an environment that runs on production settings but isn't production. > Why?

The unstated assumption of "staging" fans is that bad test coverage is a universal, naturally-occurring condition. At the companies where I've worked that had good tests, they did not have staging. At companies where I thought the test coverage was poor, they did have it.

The obvious answer is so you can test infrastructure changes and data migrations without impacting users.

In ye olde times that was the intent of "staging." Deploys were more expensive then. That's back when we talked about having some number of 9s instead of MTTR.