And like you, we have both "preview environments" and staging and for the exact reason you've highlight. I'm rather skeptical of the company behind this product now since they don't seem to understand the difference and really overselling something that's quite common.
This isn't it, and I think it's pretty much an impossible problem to solve without an entirely new engineering culture around testing.
How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc. At a company with 1000+ microservices?
And how do you populate it with actual data so you can log in and perform actions? Every single microservice probably needs test data, and it would take a lot of engineering forethought to populate the entire graph.
It gets gnarly. I've been on teams at the center of it all - user accounts and login, representation of the core business entities that the rest of the company is built upon. We couldn't even solve the problem for ourselves. There were so many permutations of account states, creating a dedicated API to make test data deviates from a first class API. So how do you coordinate with downstream services to skip IDV (matching SSN), compromised password checks, etc.
Perhaps you only bring online a subset of services and fake out the rest. At what level can you fake stuff out? And how do you ensure that the test data works with whatever subset you don't fake? If a single service responds in an unexpected way, it could break or corrupt the state you're relying upon to test.
Maybe "preview" works for frontend. Backend, not so much. This is such a massive problem, and I'm only scratching the surface with my description of the kinds of issues you have to solve.
Yeah this is the surprising thing to me - I've worked with otherwise bright people but whenever we've introduced services/microservices, it seems the default reaction is to just make a "distributed monolith" with circular dependencies and tight coupling.
I'm only half joking that 90% of my "architect" role is just point out we shouldn't have circular dependencies.
…and with added network calls!
> Microservices. grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too. seem very confusing to grug
(Source: https://grugbrain.dev/)
At a minimum, the product owner should be able to try out the new feature in the UI without having to deal with a staging environment.
For those, like me, who had to think for a moment where they had heard that reference: https://www.youtube.com/watch?v=y8OnoxKotPQ
I think what made it quite easy for us is that we maintain a staging environment. Staging is constantly tested by QA simulating our workflows so non-production data is being constantly generated. When we spin up our preview environments, we use snapshots from the staging systems to seed the preview environments.
Your point on data though really gets to the crux of the problem. Deploying the services is trivial these days with k8s, Cloud Formation, Terraform, Docker, etc. but the data part is hard.
I'd love to learn more about the challenges you have with data. My company essentially provides data-anonymization-as-a-service and we integrate with ephemeral environments, but we could probably improve the integrations.
I'll try to simplify what I mean by that but something akin to having Services A, B, and C.
Each of A, B, and C need their own definitions to define how they need to be spun up as well as a definition on how to populate their data. That data population, say through RDS, needs to be spin up RDS instances and make them ready and available whenever someone creates a corresponding ephemeral version of a service. Toss the database afterwards.
In the above case, assuming you needed a full graph, you would spin up the three services as well as attach each service to its corresponding database. This approach works well when the number of services remains small.
Once that number starts growing, it may not be cost effective to spin everything up. Say service N is needed, but you really only need to read from it. In this case, you might have a dedicated staging environment for N and create a dependency graph that says "in my ephemeral environment, create new instances of A, B, and C with their databases, but also allow network requests to make their way to N."
That might be over simplifying, but I've been working in this space for the past few years and have been thinking about this exact issue recently
Before: Your developers have a shared Linux box that serves the preview environment. Your ruby app has a tiny loader in front of it that loads the app code from a specific directory based on subdomain.
After: You need to route subdomains to different kubernetes containers, and handle deploying new containers when the developer opens a PR (because it's much more work than just copying files onto a server), and you need to handle destroying old containers (because it's much more work than just deleting files on a server).
Not to mention, usually kubernetes is managed by "the ops" who don't know anything about the app. Much more difficult to interface with them when it's not "oh one server, and the developers figure it out, and if they royally fuck it up we'll rescue them".
Routing domains to a container shouldn’t be significantly different to doing it in production.