Hacker News Reader

Comment by nickelpro | Hacker News Reader

This article has some very weird trade-offs.

They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso.

They're not arguing that testing or staging environments are bad, they're just saying their organization couldn't manage to get them working. If they didn't hit those roadblocks in managing their staging environments, presumably they would be using them.

View on HN

> They're not arguing that testing or staging environments are bad, they're just saying their organization couldn't manage to get them working.

That is exactly what I got from reading this article. Their staging process was poorly set up and they simply abandoned ship. Additionally, I was getting poor software culture vibes.

Indeed, I found the "We only merge code that is ready to go live" part odd. It seems unrelated to the presence of absence of a staging environment. Where I work, we use staging and also only merge code that is ready to go live.

Similarly, "Poor ownership of changes" and "People mistakenly let process replace accountability" just don't seem staging-related to me. I've been in environments where people throw code over the fence straight into production.

The way I read that is, if you have a staging environment, it is someone else's job to test things before they deploy (Product Manager etc.) which allows them to merge it and forget it. I agree if it is the dev's job then it wouldn't make sense.

The other issue that is fair is the potential lag between dev and production if you have a gate at staging. This way, a developer is likely to move onto something else instead of watching their baby swim into production with all the errors that could cause!

Why would you merge code that wasn’t ready to go live?

I actually empathize with the statements in the article about the challenges of managing a staging environment. For a lot of systems I worked on, short of just copying customer data into the staging environment (which, depending on your industry or the contracts with your customers may be a big no-no, and the more-distributed your system, the harder that also was to do) it was extremely hard to populate the staging environment with representative data. Or representative state in third-party systems.

(Not to say that testing on a developer's machine wouldn't have these same problems, of course, which I also find the article glosses over.)

The parity part is especially confusing to me, since they go on to test locally. So staging isn't close enough to production, but local is...?

I don't represent the original author. But the way I read that is: the staging environment is said to be a production clone, but it turned out it wasn't, so let's not pretend it is (and fool ourselves) and instead embrace a different test strategy altogether. Thinking of how I test things locally and how I write my own unit and integration tests, I guess that it means doing very isolated, functional tests. An investment in which, to some teams, may be more valuable than throwing it into staging and hoping that the state you found your staging environment in is going to surface some insight about your code change that isn't covered by your test suite.

I agree that that is what they are trying to do, but they don't appear to be testing with all that different of a strategy; something has been removed, but it doesn't seem like anything has taken it's place.

Staging is useful because you cannot predict how your changes will impact the entire application, or how it will interact with configuration. This is precisely where isolated tests fall short. Differences between production and staging are real, but differences between production and local are much more profound.

Staging is an imperfect strategy, but this approach doesn't appear to be sound. It seems to shrug it's shoulders and settle for something even worse. I'm baffled, frankly.

Having staging always encourages this. It’s really difficult to replicate prod in any non trivial way that exceeds what can be created on a workstation.

Eg. Even if you buy the same hardware you can’t replicate production load anyway because it’s not being used by 5 million people concurrently. Your cache access patterns aren’t the same, etc.

It’s far better to have a fast path to prod than a staging environment in my opinion.

I think it's too much to expect staging to match the load and access patterns of your prod system.

I find staging to be very useful. In various teams I have been a part of, I have seen the following productive use cases for staging

1. Extended development environment - If you use a micro-services or serverless architecture, it becomes really useful to do end-to-end tests of your code on staging. Docker helps locally, but unless you have a $4,000 laptop, the dev experience becomes very poor.

2. User acceptance testing - Generally performed by QAs, PMs or some other businessy folks. This becomes very important for teams that serve a small number of customer who write big checks.

3. Legacy enterprise teams - Very large corporations in which software does not drive revenue directly, but high quality software drives a competitive advantage. Insurance companies are an example. These folks have a much lower tolerance for shipping software that doesn't work exactly right for customers.

> I think it's too much to expect staging to match the load and access patterns of your prod system.

For a lot of things, this makes staging useless, or worse. When production falls over, but it worked in staging, then staging gave unwarranted confidence. When you push to production without staging, you know there's danger.

That said, for changes that don't affect stability (which can sometimes be hard to tell), staging can be useful. And I don't disagree with a staging environment for your usecases.

> For a lot of things, this makes staging useless, or worse.

That depends on what Staging is used for, if its used to run e2e tests, giving a demo to PMs etc, you can use Staging. For performance testing you can setup a similar env like Prod, run your perf tests and then kill the perf env or you can scale up the staging env, dont let anyone use it except for performance and then scale it down.

dev workstations should cost at least $4000.

Like how much productivity is being wasted because their machine is slow.

$4000 workstations are cheap compared to staging.

Multiply that by the number of employees who need it ... staging is cheaper.

Multiply that by the lost hours waiting for compilation, etc and whether you have staging or not doesn’t matter.

It’s crazy sometimes how big of a difference it is. One recent example - I had to build a custom Docker image of some OSS project. Not even a huge one - only what I would call small-mid size. Just clone the repo and run the makefile, super simple. It took 35 minutes to build on my 2020 Mac Mini (Intel) and would have been probably half that if I had the most recent machine.

Why would I build on a local machine vs running the build on a server in a datacenter? Per your own arguments, server grade hardware is going to compile much faster than any local workstation.

That’s not really true, because of Amdahl’s law you need good single thread perf in addition to a reasonable amount of cores.

Often workstation class processors, like an i9 are the sweet spot, especially with ccache as only one file is being recompiled.

Servers often sacrifice single thread performance for many more cores.

Compile time might be fractionally faster, but the latency of the CI pipeline gets you.

Ah, good old "compiling" [0]. When a worker needs a $4000 machine to actually do his work then it's unavoidable. The slow machine? $2000 out to be enough™ for everyone else.

[0] https://xkcd.com/303/

when I worked for big corp, the reason we were told in engineering for getting $1,000 laptops was that it wasn't fair to accounting, HR, etc for us to have better machines. In the past people from these departments complained quite a bit.

The official reason (which was BS) was "to simplify IT's job by only having to support one model"

Who cares what is "fair"? A decision like that should be based on an elementary productivity calculation. If not the inmates have taken over the asylum.

Perhaps we have different ideas about what a staging environment is for. I wouldn't expect a staging environment to give accurate performance numbers for a change, the only solution to that is instrumenting the production environment.

Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...

If you're choosing to pay large sums of money for SQL Server instead of the open source alternatives, you should also factor in the large sums of money to have good development/staging environments too.

All the more reason to just use Postgres or MySQL.

EDIT: as someone else hinted at, it does look like the free Developer version of SQL Server is fully featured and licensed for use in any non-prod environment, which seems reasonable.

Sure different planning 20 years ago would have made a big difference. Or the will/resources to transition. I am just saying that this scenario exists.

Non-prod is free.

The developer version, yes. But I have not seen the AWS amis for the developer version: https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-ec...

You can't install the enterprise non-prod for free. (But the developer version is supposed to have all the features)

It's pretty easy to create your own amis with developer versions. It makes sense why AWS doesn't necessarily provide this out of the box. But it still stands for fully managed versions of licensed software, you'll pay for the license even if it's non-prod

Yes, that's not to say it is not possible to create a similar env, but I thought the debate was how precisely you are replicating your production env.

Sure it may be "good enough", but I thought the debate was about precision. How your own ami setup may differ from the AWS built from the developer version compared to the AWS ami? I don't know.

Trying for an identical setup in staging is expensive, this is just a scenario I am familiar with. I am sure there are a lot like this.

I was thinking about this line from the article:

> More often than not, each environment uses different hardware, configurations, and software versions.

They can't even deploy the same software versions to their staging environment. We're a long way off talking about precisely replicating load characteristics

You could call that an infrastructure problem. You have built an expensive infrastructure which you cannot afford to scale to the extent you desire.

> Depending on your tech, staging environments can be very expensive

For our business & customers, a new staging environment means another phone call to IBM and a ~6 month wait before someone even begins to talk about how much money its gonna cost.

That's per server, not per database. I have dozens of staging databases on my sql server - product_name_pr_id.

Exactly. “Staging never matches Prod” - well why is that? Make it so!!

I have never ever even heard of a place where that was possible.

The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.

I worked at a financial company that had an exact copy of its largest datacenter deployment in a lab for full-system testing. They recorded packets from the exchange at nanosecond precision, and had the equipment to do exact playback of those packets in the lab.

It was an amazing engineering tool. You could test all sorts of end-to-end performance tweaks there, and have great confidence that you were right.

I also worked at a financial company that did not have one of these. The computers ran substantially slower, and lots of end-to-end performance improvements were left on the table and mired in debate. The mock datacenter cost about $10 million, and was worth every penny.

I am curios, why do you think it's impossible?

I think we can establish that the database is the biggest culprit in making this difficult.

As an independent developer, I have seen several teams that either back sync the prod db into the staging db OR capture known edge cases through diligent use of fixtures.

I am not trying to counter your point necessarily, but just trying to understand your POV. Very possible that, in my limited experience, I haven't come across all the problems around this domain.

The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors.

There's a certain amount of "best practices" effort you can go through in order to make your preprod environments sufficiently prod like but scaled down, with real data in their databases, running all the correct services, you can have a load testing environment where you hit one front end with a replay of real load taking from prod logs to look for perf regressions, etc. But ultimately time is better spent using feature flags and one box tests in prod rather than going down the rabbit hole of trying to simulate packet-level network failures in your preprod environment to try to make it look as prodlike as possible (although if you're writing your own distributed database you should probably be doing that kind of fault injection, but then you probably work somewhere FAANG scale, or you've made a potentially fatal NIH/DIY mistake).

The article doesn't talk about any of that though. The article says staging diffs prod because of:

> different hardware, configurations, and software versions

The hardware might be hard or expensive to get an exact match for in staging (but also, your stack shouldn't be hyper fragile to hardware changes). The latter two are totally solvable problems

With modern cloud computing and containerization, it feels like it has never been easier to get this right. Start up exactly the same container/config you use for production on the same cloud service. It should run acceptably similar to the real thing. Real problem is the lack of users/usage.

I was responding to other commentors not really the title article.

The stuff you cite there is pretty simple to deal with, configuration management is basically a solved problem and IDK how you can't just fix the different hardware.

The more universal problem of making preprod look just like prod so that you have 100% confidence in a rollout without any of the testing-in-prod patterns (feature flags, smoke tests, odd/even rollouts, etc) is not very solvable though.

A lot of things seem like they shouldn’t be, until you’ve debugged a weird kernel bug or driver issue that causes the kind of one-off flakiness that becomes a huge issue at scale.

IME, when you are not webscale, the issues you will miss from not testing in staging are bigger than the other way round. But that doesn't mean that all the extra efforts you have to put in the "test in prod only" scenario should not be put even when you do have a staging env.

As if this wasn't enough of a headache, GDPR regulation requires more safeguards before you can put your prod-data in a secured staging environment.

Then there is the database size, which can make it hard and expensive to keep preprod up to date.

And should you want to measure performance, then no one else can use preprod while that is going on.

It is possible.

But you need infrastructure and paying delicate attention to this problem. It is hard to define exactly what does replicating prod mean. And sometimes it might be difficult, e.g. prod might have access controlled customer data store that has its own problem, or it is about cost. But doesn't necessarily mean if you can't replicate perfectly, it is useless, you can still catch problems with things that you can replicate and do go wrong.

Ofc it is impossible to catch bugs 100% with staging, however, that argument goes either way.

I've worked at a number of places where we had multiple products with replicated staging/production environments. One approach is to have your environments codified, prod gets built from the same pipeline staging does, and to automate database refreshes from prod > staging so they don't fall behind. It isn't rocket science, but of course it doesn't come at zero cost. Some production environments are pretty hard to replicate too, like anything with third party integrations that don't offer a staging environment of their own.

If you're a small shop I can understand it, but bigger companies with infrastructure teams, there's no excuse really, the technologies are all there.

> I have never ever even heard of a place where that was possible.

You set the CI/CD pipeline to enforce that deploys happen to staging, and then happen to production. That's it. It's not hard.

Tech debt can definitely accrue that makes it difficult. For instance recently I had an odd difference between prod and staging, and when I looked into it I realized there was a legacy behavior for old customers, and our testing users in different environments were on different sides of the cutoff.

Naturally that's a nasty tech debt and we should find a way to clean it up, but it was a pragmatic solution to a business problem that occurred before any of my team mates started working there, and it meant we could continue to serve our customers as we made a transition. These things happen.

Found the guy thats only worked at a company of 1000 people

For many, many cases, it is literally impossible, if not incredibly unrealistic. Doing away with it eliminates all of the problems associated with it and allows you to gain more confidence in your deploys. It simplifies while adding reliability and velocity. This is why abandoning staging is the best strategy.

If that's how it is at every single company then saying "just make it the same" probably isn't the answer.

Reply on news.ycombinator.com