Hacker News Reader

Comment by dehrmann | Hacker News Reader

I have two. One was caused by data inconsistency between services and regions. One is more hypothetical: the microservices had gotten to the point that no one knew how to start the system if all services are down, and it's possible that services have circular dependencies to the point that it would be incredibly hard to do a cold start.

View on HN

I've actually seen your hypothetical in action, but the bug was even more subtle. Assume service A, B and C. A and C both need information from each other which is usually cached. Normally, you'd deploy one service at a time so the call chain would go A -> B -> C -> A or A -> C then A -> B -> C but in this particular instance, A and C's caches were cold, causing an explosion of service calls that took both services down.

> A and C both need information from each other

Sounds like a monolith pulled apart :-)

You handle the hypothetical by validating a new environment can be built and bootstrapped. Doing this on a regular basis, either by tearing down and rebuilding dev, or in a separate environment just for this purpose, is not a "nice to have". This same problem exists with monoliths with complicated dependencies, nothing new here.

Just like backups, if infrastructure as code isn't tested, it's worthless.

I think that if there is a genuine circular dependency then the services won't start ever. But I think it is possible to introduce services that assume other services are up and have an apparent dependency circularity. The trick is to have all your services resilient to it's start requirements not being met - basically the service has to back off and wait if information it needs isn't yet in the environment... and then ask again - so that when other services are up everything syncs and comes up.

That's fine, until the environment doesn't come up because a hundred services all queried the source of truth simultaneously and caused load that prevented any service from getting its config

Which is easy to deal with by introducing a random factor into the backoff.

> One is more hypothetical: the microservices had gotten to the point that no one knew how to start the system if all services are down,

this is a good one and have been thinking about this myself. Even with smallish projects that might have 10s of container based applications / services. In my case I end up with what is essentially a 3 tier architecture with each tier being a group of containers/machines with their own rules for startup/shutdown.

I’d think the best way to handle a cold start is to have each microservice fail fast and be wrapped in a supervisor. They will converge to uptime.

Unless you have a truly circular dependency, at which point they probably should be collapsed into a single service.

Reply on news.ycombinator.com