Comment by wgerard | Hacker News Reader

Comment by wgerard | Hacker News Reader

Cool article, enjoy the summary of relevant knowledge that's been passed around various circles.

I do disagree with:

> Environments like staging or pre-prod are a fucking lie.

You need an environment that runs on production settings but isn't production. Setting up an environment that ideally has read-only access to production data has saved a huge number of bugs from reaching customers, at least IME.

There's just so many classes of bugs that are easily caught by some sort of pre-prod environment, including stupid things like "I marked this dependency as development-only but actually it's needed in production as well".

Development environments are frequently so incredibly far removed from production environments that some sort of intermediary between production is almost always so helpful to me that the extra work involved in maintaining that staging environment is well worth it.

It's not the same as production obviously, but it's a LOT closer than development.

> You need an environment that runs on production settings but isn't production.

Why?

> Setting up an environment that ideally has read-only access to production data has saved a huge number of bugs from reaching customers, at least IME.

That's an anecdote, not a reason. Also, just because you've done it that way doesn't mean it has to be done that way, like you asserted.

> There's just so many classes of bugs that are easily caught by some sort of pre-prod environment

Also does not support the claim that you need a pre-prod env.

> Development environments

Whoa, there! You're sneaking yet another kind of environment into the conversation? Maybe not. This is unclear, given the many different ways that people do work.

> not the same as production obviously, but it's a LOT closer

You seem to want something like production. There is nothing more like production than production.

If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way.

The obvious answer is so you can test infrastructure changes and data migrations without impacting users.

In ye olde times that was the intent of "staging." Deploys were more expensive then. That's back when we talked about having some number of 9s instead of MTTR.

> If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way.

You seem to be assuming 1. some sort of large-horizontal-scale production system with multiple customers, where the impact of a failure can be minimized by minimizing the number of users exposed to new features, and where 2. there's no type of bug in the code that would potentially take down production as a whole.

What if your production system is, say, a bank's ACH reconciliation logic? A medical device? A car? The live server for a popular MMORPG? A telephone backbone switch? A television or radio broadcast station?

In these cases, your software isn't a service with multiple distinct customers that each make requests to it, where you can test your new code on one customer in a thousand; your software is just running and doing something—one, unified something per instance of the system (though that process may track multiple customers)—and if the code is wrong, then the whole system the software operates will fail.

How do you test software for such systems?

Usually by having a "production simulation" whose failure won't kill people or cost a million dollars in lost revenue.

Thank you for contrasting life building the latest social media platform from what many of the rest of us do.

Currently I work on systems to prepare and validate birth and death certificates for the state, counties, hospitals, et al, and this whole “throw it against the wall and see what sticks” methodology doesn’t fly. Nor would it have worked when preparing and presenting investment account information 5 years ago, nor the job 10 years ago processing lawsuit and insurance claim cases and legal bills. Nor any place that I at least have ever worked in the last 30 years.

Agree, and I work in the latest 'social media platform' type end. We have many customers. I can assure the author of the post, when those customers pay for enterprise licensing and their system is broken with an obvious bug. The 'we didn't do any testing before hand because staging is a lie' doesn't actually fair well as a valid excuse for anything. In fact, you just look like an unprepared and immature muppet.

Well in that case you're not talking about the original scenario where we were talking about whether or not to use a pre-production setup that mimicked production as closely as possible, are you?

> If you're set up to do A/B tests or deploys with canaries or give potential customers test accounts you're probably able to start testing in production in a sane, contained way.

Basically you're outsourcing QA to your customer. Some systems may afford this, others not.

> Basically you're outsourcing QA to your customer.

You've just described any software ever used.

Not in any meaningful way is that true, no.

>> You need an environment that runs on production settings but isn't production. > Why?

The unstated assumption of "staging" fans is that bad test coverage is a universal, naturally-occurring condition. At the companies where I've worked that had good tests, they did not have staging. At companies where I thought the test coverage was poor, they did have it.

Reply on news.ycombinator.com