Hacker News Reader

Comment by linsomniac | Hacker News Reader

If you spend $100K on a staging environment and it doesn't give you confidence that it won't fail on production, the problem is Your Staging, not The Staging.

If we have something fail in production, the first question we ask ourselves is: Why didn't we catch this in staging? 90% of the time, the answer is: "We didn't test if properly but thought we did."

Our rate of patching for defect is rare.

As far as the cost, our staging is run in hardware retired from production, with some upgrades (mostly RAM and SSDs). I'm not sure we've spent the authors $100K over 2 decades, let alone in a single year. But YMMV.

View on HN

Everybody is in the cloud now, where there are no hand-me-downs, or at least none we get to use.

A year and a half ago, IBM did a study and found that most companies are only 20 percent to cloud transformation. The reality is that there is a ridiculous amount of private datacenters and iron around.

https://www.ibm.com/blogs/cloud-computing/2019/03/05/20-perc...

This makes it sound like 100% is the goal. Some companies don't get added value by going to the cloud. The cloud isn't cheap for example (among many others) for storage heavy workloads. And sometimes it's also a regulatory thing.

Thank you for saying this, I am more than frustrated by that mentality whenever I go to conferences or message boards. There’s a big world out there and most of does not resemble the practices of internet companies. It’s this fallacy of inevitability for cloud.

When AWS releases a new instance generation, everyone flocks to them. For previous generations theres usually ample capacity to run as spot instances uninterrupted over long periods of time, giving your staging environment significant savings. I'd call that a hand me down.

Except the top level comment was doubting they ever spent 100k total in the history of their staging system. That’s not going to be true for AWS even with the nosebleed seats, is it?

And are you really going to run your entire staging environment in spot instances? What happens on peak days?

I didn't mean to imply that the author couldn't have spent $100K. There are clearly situations where an expensive staging environment would be needed. I was just saying that we've been able to do our staging for a fraction of that price, and had good results.

Once I visited a customer site who were having a performance problem. They ran everything on AWS. The day before I arrived on-site, they spun up a brand new production-sized test environment just for me to debug their issue in. I spent a day or two on-site working out what was causing their performance problem. When I left, they deleted that environment. They only paid for its existence for 2-3 days. So with cloud you can get away from fixed always-on staging environments, and move to "staging environments on demand".

(My recollection was they loved everything about AWS except the price tag, although creating a personal staging environment for me to debug their issue in would have only made a marginal difference to that total cost.)

Reply on news.ycombinator.com