How we keep dependencies fresh across 45+ microservices

medium.com

14 pointsavitzurel3 comments

how do you handle when you major version upgrade to your microservice that breaks 7 other microservices in the CI, do you fix it? Do they fix it?

I can give you one example where something like that happened.

All of our services use credstash to store secrets. We use KMS in order to encrypt, decrypt the secrets and when the app loads we load all the secrets into memory.

There was a major change to cryptography that caused credstash to break and in turn break a large set of our services.

This PR [1] was submitted by one of the engineers to force the lower version to credstash and we opened the dependencies again to reload credstash on all services.

The process to track it down and fix it took about an hour and all services were building correctly again.

We make a very conscious choice to break those things in real time and deal with them rather than hiding them behind locked dependencies you simply never upgrade.

[1] https://github.com/fugue/credstash/pull/174

We also try to avoid the you/they dichotomy. It's bad culture.

Reply on news.ycombinator.com