Hacker News Reader

Comment by giancarlostoro | Hacker News Reader

In a high quality setup you have a staging server that is a carbon copy of PROD. Bonus points if you make it so staging and PROD are 100% interchangeable, to the level that you can point PROD to staging, and then turn PROD into staging, and do the same next deployment. If you can do that, you have a stronger change of at least reproducing production issues.

Dev, UAT / QA, Staging, PROD. This is the ideal setup in my eyes. It lets QA / UAT hold changes that are maybe not 100% ready, while not blocking testing that is mean to go into PROD ASAP because it can sit in staging.

View on HN

There are many setups where this is not just not possible. In some cases, doing this is prohibitive because of cost or prohibited by law.

+ for case of cost: lots of very large companies have prod environments that cost big $$$. Business will not double prod cost for a staging environment mirroring prod. Take an example of any large bank you know. The online banking platform will cost tens if not hundreds of millions of dollars to run. Now consider that the bank will have hundreds of different platforms. It is just not economically feasible.

+ for the case of law: in some sectors, by law, only workers with "need to know" can access data. Any dev environment data cannot, by law, be a copy of prod. It has to be test data, even anonymization prod data is not allowed in dev/test because of de-anonymization risk.

Given this, consider a platform / app that is multi-tenant (and therefore data driven ) eg a SaaS app in a legally regulated industry such as banking or health care. Or even something like Shopify or GMail for corporate where the app hosts multiple organizations and the org to be used is picked based on data (user login credentials).

The app in this scenario is driven by data parameterization - the client site and content are data driven e.g. when clientXYZ logs on, the site becomes https://clientXYZ.yourAppName.com and all data, config etc are "clientXYZ" specific. And you have hundreds or thousands of clentsAAA through clientZZZ on this platform.

In such a world, dev & test environments can never be matched with prod. Further, the behaviour of the client specific sites could be different even with the same code because data parameters drive app behaviour.

Long story short, mirroring staging and prod is just not feasible in large corporate tech

To be any use staging environments should be scientific tests. They should prove that a given change, if it goes to production, will work.

You cannot do this if you're changing more than that one thing. The only way to make this work really is either dynamic environments that completely mirror everything, which tends to be time consuming or expensive or continuous delivery to a production-like environment via feature flags and so forth.

Having a staging server that is a mirror of production[1] improves things a bit over doing nothing. You need the entire environment, including all your dependencies, to have a real test of anything, and that includes things that corporate IT departments typically hate.

[1]: Why is it so common to see "PROD" written as if it were an acronym?

I always write it that way maybe for the same reason others do it, to emphasize how critical PROD is, so you don't overlook it if you just read prod, or production. If you see PRODUCTION you might slow down and go "oh crap" so it is definitely an emphasis I always add when talking about production in text. PROD is just shorter to write, but all caps makes the emphasis stick.

If you staging environment is pointing to the exact same databases PROD is, and other similar dependencies, there's no reason you can't hotswap it with PROD itself, I mean I've done something like this before.

It's much easier if your production deployment pipeline is setup for it though. You'd want to scale down drastically for staging, but in my eyes, if you're not going to have staging be as carbon copy of PROD as you humanely can have it, you might as well not have that fourth environment and just suffer when you cannot reproduce bugs. The real gem of staging is that if it would break in PROD, it would definitely break in staging. In the few companies where we had a carbon copy of PROD setup as a staging environment where key things are pulled from PROD itself, we've had way less bugs promoted to PROD when QA tests them in staging.

In theory the ROI is worth it, if you care about quality. Sadly most places do not care about quality nearly enough.

But it makes the text look like it was written by a schizophrenic: <https://web.archive.org/web/20231122160401/https://prestersp...>

I guess, but its consistently the same word being capitalized.

>In a high quality setup you have a staging server that is a carbon copy of PROD

In low throughput environments I see stuff like this. The problem is with high throughput environments it doesn't tend to happen because of the massive expense incurred.

just saw your answer - we are thinking exactly the same thing but I took the long-winded route to saying it

Today in petty off-topic complaints I expect to burn some karma on: PROD isn't capitalised, is an abbreviation of Production, not an initialism of Public Ready Outside-world Delivery.

PROD isn't capitalized because it's an initialism. It's capitalized because the machine is screaming at you that this is production, be careful. ;)

Reply on news.ycombinator.com