Hacker News Reader

Yeah. And it's also not that impressive. We've had what they call "preview environments" for a very long time and at my previous companies too. With GitHub and AWS/Cloud it's trivially easy to do.

And like you, we have both "preview environments" and staging and for the exact reason you've highlight. I'm rather skeptical of the company behind this product now since they don't seem to understand the difference and really overselling something that's quite common.

Yes, this is one of those things that was easier before kubernetes.

Before: Your developers have a shared Linux box that serves the preview environment. Your ruby app has a tiny loader in front of it that loads the app code from a specific directory based on subdomain.

After: You need to route subdomains to different kubernetes containers, and handle deploying new containers when the developer opens a PR (because it's much more work than just copying files onto a server), and you need to handle destroying old containers (because it's much more work than just deleting files on a server).

Not to mention, usually kubernetes is managed by "the ops" who don't know anything about the app. Much more difficult to interface with them when it's not "oh one server, and the developers figure it out, and if they royally fuck it up we'll rescue them".

It’s really not that difficult to automate it over helm or something. All you need is a registry, and most cloud services provide a registry, even GitHub has one.

Routing domains to a container shouldn’t be significantly different to doing it in production.

For that matter, you generally will have a single app firewall or ingress server that reverse-proxies to your application anyway.

Helm, registry, firewall, ingress, reverse proxy, these are all concepts that whoever sets up this hypothetical staging environment will have to know about, and will also have to be a developer (in order to care). So basically you need a senior dev to either love kubernetes or learn enough of it to get the job done. This is what I mean by "harder".

I have spent a lot of time thinking about this problem. I would gladly pay for a product that could make a real, honest to god preview environment that you could spin up on-demand and populate it with all of the data you need.

This isn't it, and I think it's pretty much an impossible problem to solve without an entirely new engineering culture around testing.

How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc. At a company with 1000+ microservices?

And how do you populate it with actual data so you can log in and perform actions? Every single microservice probably needs test data, and it would take a lot of engineering forethought to populate the entire graph.

It gets gnarly. I've been on teams at the center of it all - user accounts and login, representation of the core business entities that the rest of the company is built upon. We couldn't even solve the problem for ourselves. There were so many permutations of account states, creating a dedicated API to make test data deviates from a first class API. So how do you coordinate with downstream services to skip IDV (matching SSN), compromised password checks, etc.

Perhaps you only bring online a subset of services and fake out the rest. At what level can you fake stuff out? And how do you ensure that the test data works with whatever subset you don't fake? If a single service responds in an unexpected way, it could break or corrupt the state you're relying upon to test.

Maybe "preview" works for frontend. Backend, not so much. This is such a massive problem, and I'm only scratching the surface with my description of the kinds of issues you have to solve.

Microservices became this mess because people only think about production. Nobody builds microservices as standalone units that should be testable and runnable in isolation, with ZERO external data. Instead, everything relies on everything, so it's just a monolith split up into microrepos. Worse, people have started putting app configuration in S3 and other dumb things (I've seen it...) do you cannot run things locally, or even match production unless you have AWS creds. Madness.

Microservices should indeed be able to run in isolation, but that doesn't remove the need for end-to-end and integration testing.

At a minimum, the product owner should be able to try out the new feature in the UI without having to deal with a staging environment.

> Nobody builds microservices as standalone units that should be testable and runnable in isolation, with ZERO external data. Instead, everything relies on everything, so it's just a monolith split up into microrepos.

Yeah this is the surprising thing to me - I've worked with otherwise bright people but whenever we've introduced services/microservices, it seems the default reaction is to just make a "distributed monolith" with circular dependencies and tight coupling.

I'm only half joking that 90% of my "architect" role is just point out we shouldn't have circular dependencies.

> Instead, everything relies on everything, so it's just a monolith split up into microrepos

…and with added network calls!

> Microservices. grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too. seem very confusing to grug

(Source: https://grugbrain.dev/)

The way I originally did it at my previous company was to have each service have a "dev" Flask blueprint that got registered on non-production environments, that had an endpoint that could populate the database with sample data, and a coordinating service to populate them in the right order (so IDs matched up). Worked pretty well.

Sounds like a large dependency graph and multiple variants of spinning up the environments.

I'll try to simplify what I mean by that but something akin to having Services A, B, and C.

Each of A, B, and C need their own definitions to define how they need to be spun up as well as a definition on how to populate their data. That data population, say through RDS, needs to be spin up RDS instances and make them ready and available whenever someone creates a corresponding ephemeral version of a service. Toss the database afterwards.

In the above case, assuming you needed a full graph, you would spin up the three services as well as attach each service to its corresponding database. This approach works well when the number of services remains small.

Once that number starts growing, it may not be cost effective to spin everything up. Say service N is needed, but you really only need to read from it. In this case, you might have a dedicated staging environment for N and create a dependency graph that says "in my ephemeral environment, create new instances of A, B, and C with their databases, but also allow network requests to make their way to N."

That might be over simplifying, but I've been working in this space for the past few years and have been thinking about this exact issue recently

While the problem is still hard, it is definitely easier to solve with fewer, larger services. If your backend consists of 3-5 services, the problem is far more tractable than if it consists of 1000+ services.

Yeah that's a really good point. Our microservices are like 30? 1000+ is intractable but up to 50 or even a 100 it's doable. We rely pretty heavily on AWS Cloudformation to deploy. All our services have Cloudformation templates. I'm sure Terraform would work too or be even better. Since every service knows how to deploy itself, building a preview environment wasn't too difficult. It's just a matter of stitching them all together, which was made a lot easier with Cloudformation nesting of stacks.

That's a really good point.

I think what made it quite easy for us is that we maintain a staging environment. Staging is constantly tested by QA simulating our workflows so non-production data is being constantly generated. When we spin up our preview environments, we use snapshots from the staging systems to seed the preview environments.

Your point on data though really gets to the crux of the problem. Deploying the services is trivial these days with k8s, Cloud Formation, Terraform, Docker, etc. but the data part is hard.

> but the data part is hard.

I'd love to learn more about the challenges you have with data. My company essentially provides data-anonymization-as-a-service and we integrate with ephemeral environments, but we could probably improve the integrations.

Got a link to your company's service? Would love to learn more. We work in healthcare so data-anonymization is essential. If we can take prod data and erase all the PII, it would be AMAZING.

> How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc.

For those, like me, who had to think for a moment where they had heard that reference: https://www.youtube.com/watch?v=y8OnoxKotPQ

The reason every complex application I've worked on has had a staging environment is because you do need to test production deploys in an environment that mirrors production dataset and infrastructure. Especially with data migrations, distributed databases. That is prohibitively expensive and not feasible to run in n+1 envs.

Big if, but if you can use database containers it's relatively low cost to spin them up in a namespace, load data from a database snapshot, run the migration, and then tear down the container or even the entire namespace.

> If you were to get rid of staging would you stand up your entire backend every time you made a preview environment? Does that include replicating databases or would you just use seed data that may or may not resemble what's in production?

What I've seen done is that the staging environment is skipped, but instead concentric releases are performed on the customers. So released code goes first to a small group of users, and then widens from there if everything looks good.

This technique doesn't seem especially relevant for many kinds of backend work.

> What I've seen done is that the staging environment is skipped, but instead concentric releases are performed on the customers. So released code goes first to a small group of users, and then widens from there if everything looks good.

If you really want to invest in extensive testing, you can do both! Hell, _and_ you can have a load testing environment as well.

Yeah, we had tooling to load a snapshot of customer data (with PII removed) into an ephemeral environment for when developers needed data that was more authentic. This worked for performance related work or just for features that needed a complex data model to be useful (ML apps, etc)

Schema changes are hard to roll out that way though.

We were a B2B product whose multitenancy model was schema-per-customer, so it was no problem for us.

Agree that it would be harder to implement with customers sharing a schema, but you can still have different release pods

How long do your PRs stay open? Could you explain a bit more about what happens between opening the PR and merging it, like automated/manual testing? At my company we do something similar, except we trigger production deployments off tags, not merged PRs. Merging goes to staging only.

It really depends on the PR. Most do get merged fairly quickly (same day) but some where there's more discussion can stay open for several days or even a week.

All of our codebases have some level of automated testing that runs when the PR is created. That could be unit tests of a single function or Playwright tests which exercise an UI flow. What those tests are dependent on the type of software and the team building it.

The preview environment links that are generated when the PR is open are given to QA, Design, and Product to take a look at. Not every one of those roles review every PR, it really depends on the task. But if their feedback is required, they'll give it at this stage.

Probably worth mentioning at this point that we have separate repos for each web application and service. It's service oriented but not microservices, services at our company encapsulate a relatively large domain and it's similar with web applications, right now each web application has its own subdomain.

So PR gets merged, the pipeline will deploy that code to staging. At this point the service owners have the option to define a set of integration tests, not every repo has them but it is an option to run them. If those tests fail we do a rollback. If those tests pass (or if they didn't have any defined) then a second deployment to production is triggered.

All of our deploys are blue-green, so no downtime except for the occasional blocking database migration in which we have to schedule downtime. User facing features on the web applications are all required to be released first behind a feature flag. For those there will be several releases behind a feature flag, then we'll flip the flag first in staging. We'll have Product and QA do a complete run through and make sure they're okay with everything, then the "real" release happen when we flip the feature flag in production (no deployment needed).

Every night we clone the production database, sanitize it (swapping out things like names, phone numbers, etc.) then point staging to that clone. Then we run an end to end regression test that goes through all of the happy path flows for all of our apps.

>a sanitized copy

Quite the importance over a constant wave of changing code on development in certain projects.

Staging seems like a more code locked stable area for all previews that can be polished via quality practices and keen eyes.

Now it seems a little fleeting to me as well to have the shock of previews enter into this instead.

Let the paint dry adequately before taking it outside.

Setting up preview environments automatically (in kubernetes or azure) with sanitized production data is something that always sounds amazing to me, but I have always had difficulty to come up with ideas to implement it. Every company/team uses different tools and ways to implement this, so there isn't a one-size-fits-all solution.

Do you have any pointers?

How do you sanitize data between prd and stg?

You can use something like this https://www.replibyte.com/docs/introduction/

My company can anonymize production data while preserving the utility of the data. You get privacy-safe data that looks just like production data with the same distributions. It's real data.

The typical workflow is to anonymize production data, snapshot it, and make it available to preview and testing environments as replicas. It's pretty fast.