We built network isolation for 1500 services
monzo.com
monzo.com
However, 1500 services? That really feels like they're separating things at too granular a level. Does every one of those things really need to sit behind a network call? Couldn't some of that re-use be via code libraries? I wonder what the service to developer ratio is?
nevertheless 1500 is a lot, but if you think it's a lot, wait till someone builds out their entire business with lambda/serveless. :-)
I’m actually not surprised you have a service for handling working days. The list of UK bank holidays should ideally be stored in one place, with everything else consuming that. But not all of the 1500 services have their own data stores, do they?
All the services have their own keyspace in Cassandra, there is near-zero cross service data access
So, what is bad about a monolith? The complexity. Coupling of unrelated things; because it's simple to do so, because it's right there next to you and you know how it's implemented and what it performs, so you do thing A in component X knowing/relying on the fact that component Y will do the thing B, because that's how it's implemented (not because it's inherent unavoidable business logic).
How do you end up with a distributed monolith? E.g. assume that microservice B will call microservice C only after you (i.e. microservice A) had a chance to call C first. Yes your code is distributed, but you don't get the benefit of logic segregation between the modules - you just get all the downsides of distributed systems.
Do note that my scenario requires no shared code or library. Not sharing code doesn't save you from the distributed monolith - careful thought does.
I've never worked on microservices, so I may just have a fundamental misunderstanding of how granular you're supposed to be & also don't know if this is the norm.
Would be nice to hear some other opinions on this.
If you update the business day library, then you also need to bump the version of this lib used in your 100+ “real services”, or risk some services seeing differences on this front.
Imagine a bug from different logic due to version differences between libs in a distributed system. Would be a mess to debug.
If you say “oh well services should just install latest” now you have the distributed monolith problem. A new version of a library could get pushed out but have a regression bringing down half your services
I fail to see how they won't have the exact same problems.
Doing the same thing with shared libraries requires redeploying everything that is dependent on the bad library (i.e. hundreds of containers instead of two or three).
It's not necessarily that one model is better than the other just that using a hybrid approach often means having the issues of both with only a few (if any!) of the benefits.
If you have to make a backwards-incompatible change, you probably need to make your service support two different API versions for a while, and hunt down services that are calling old stuff for a while.
This is basically how you can move forward on improvements without coupling "improve this microservice" to "update every single other microservice that is calling to it".
Though I believe part of the theory with microservices is that since you are working on such tiny surfaces, most services will never need to roll out a backwards incompatible change
It's actually a great example of something that changes rather frequently, and it's a concern that cuts across dozens (or in Monzo's case, maybe hundreds) of services.
Causing a roll-out to hundreds of critical business functions (whether hosted in a monolith or microservices) because Uganda added a holiday seems quite excessive.
The rule I and my company follows is that the shape of a module should follow its deployment model and scope. Some services are global in scope (literally targeted at the earth) others are scoped to a small subset of our inner network. They should have a different code repo because they have a different rollout schedule.
In the case of the working-days services. Only 1 rollout needs to occur, and it seems like a fairly safe rollout.
Let's consider another service, the CriticalTransactionService, or CTS. It executes critical business transactions in a very stateful way. When deploying a new version, in order to avoid any loss of availability, often a special rollout dance must be executed. E.g. switching the master transaction writer to another region, which means changing databases from passive replication to active master, and vice-versa.
One might consider this rollout "risky" and therefore only limits it to happen at 3am on a Saturday. It seems like a good idea to limit the scope of this 3am Saturday rollout only to the CriticalTransactionService, and most other services are free to deploy to prod whenever they wish.
Doesn't the data just live in a database & there's a cache going on that can be invalidated?
Or is everything just so overcomplicated these days?
How would you roll out this change to production? If you just swapped out tables and stored procedures, you'd have downtime. You can't just install a parallel table and stored procedures, because there is no way to tell all of the consumers to use the v2 of your functions. So you'd have to temporarily remove availability of the WorkingDays functionality.
If it were deployed as a microservice, you are free to deploy a v2 of WorkingDays in parallel. When it's live, the old one goes away, and there is no loss of availability.
Some of the things that can go wrong are:
- One or more caches failing to be invalidated by your "update business-day database" function.
- Temporary loss of connectivity to a cache from the updater, resulting in failure to proactively invalidate, so incorrect business-day results used by some services depending on which cache they read.
- Additional caches you didn't realise someone had added to the application or library code, that don't get invalidate.
- Behavioural inconsistencies as different functions in the distributed system read either the database or different caches, and get inconsistent results shortly after an update (after DB write, before proactive invalidation)
... In other words, the usual problems with distributed systems.
Also some logic issues, which a microservice is more likely to detect and log:
- No entry in the database for some far-future or far-past date, that application code assumed was ok to query, but nobody filled out in the DB.
Anyway, a database table can be thought of as a kind of microservice, in the sense that it's also remote network call, and it's also something you need to treat as an API contract with careful updates. If you aren't using a consistently distributed database, it suffers from multi-zone latency just like microservices calls.
Another reason why something like "business day" might be a function of stored data rather than just database data is that's actually a bit of a complicated and evolving logic as the product evolves:
Maybe the first application logic assumed a business day to be "Monday to Friday except for national holidays in the DB", did a DB lookup for the holiday flag, and combined with Mon-Fri. Later, we expanded to new countries and find it's different in other countries. Later, we found country was insufficient and it needed further details about administrative regions.
After each step, the previous database query and logic would still work but be incorrect due to missing a query field. So at each step, new logic needed to be rolled out globally across all services. For behaviour consistency across the system, all at the same time (similar to a DB schema change).
With a microservice, the logic update would be global and immediately consistent. Each part of the application could be updated separately, on their own development schedules, to provide the additional query field, long before the microservice was switched over to requiring the field. After that the microservice would fail any requests without the additional query field, making certain all functions and services acted consistently with the new logic, or failed.
...Having said all that, my favourite isn't microservices at all. In principle, logic updates can be rolled out with ACID-like properties and transactions, all the way to the edge of a system, just like data updates and cached data. There is no inherent reason why application and library code needs to suffer from complicated updates across a large business system. Business logic can be dynamic, replicated, and guaranteed consistent, while at the same time being available as fast, direct function calls. But I've rarely seen this implemented.
This is likely trivial for the holidays endpoint, but I'm interested in the more complex ones.
Is there a separate reason to those? Do you worry about the additional latency of additional service calls (for some trivial functions that doesn't feel terribly worthwhile)?
edit: Also, for the same reason chaos engineering, short lived certificates and periodic backup tests make sense; constantly testing out the deployment of all services makes sense to me. If you do something like deploying a service rarely then the chance of something going wrong (forgot a step, radical external changes, etc.) is decently higher.
Perhaps you could built in a "regular" fashion and just compile it like that? You could write normal code and let the compiler do the massive work to handle all that and compile everything into somethign that distributes over the network.
[1] https://twitter.com/JackKleeman/status/1190354757308862468
E.g. https://medium.com/@ahelwer/checking-firewall-equivalence-wi...
This is much more scalable in the long run.
1. Why was static analysis of the code chosen over observing the system during runtime and integration testing?
2. What was the reason rhe CNI layer was chosen for the implementation of this over the service mesh layer?
Something that really interests me about bazel/buck/pants/please is it automates #1 entirely with dep queries.
but you will have the service names that you're contacting configured somewhere (or in the worst case hardcoded). I would think this is much easier to analyze than the traffic flows.
This is just saying that writing "tests is hard". I agree, it is.
I always wonder about the value of static analysis. I've used it myself and have found bugs with it, but they are mostly easy bugs to find, and there's a lot of noise. The hard bugs are often only found with serious integration tests. I'll feel much more confident with a codebase that's 90% covered by tests than one that's statically analyzed.
Dynamic analysis, instrumentation and intelligent logging are invaluable in catching a good chunk of the remaining non-trivial errors.
(I have a soft spot in my heart for fuzzing, which really should be considered as just another form of exploratory/destructive testing.)
I actually agree. But the implication is that static analysis should really live in the IDE, not the compiler or test suite. It's best when catching the silly whoopsy bugs as you're typing them. But I've yet to see a static analyzer that can catch race conditions, network or I/O errors, or something that can verify what you expect to happen actually does happen.
Thinking about static analysis on a higher level, arguably they shouldn't be necessary at all. If a system can statically analyze your code and find faults, then it implies that there is a deficiency in the language itself.
And they also add the static analysis to their CI pipeline, so that it can be checked then and there, not by obscure "happens once every half year during runtime" integration with a very uncommon service that they don't build very often.
> We generally fixed those cases by adding a special comment in the code that told rpcmap about the link
Why not enforce all endpoints/urls be defined in a config file and sidestep this? - scanning code for URLs/constructed URL is overkill and brittle.
E.g. the monitoring service should only be able to access the metrics part of each service.
IMM, the security orchestration may actually become the "app" as speeds continue to increase, compute costs go even lower and losses incurred from compromised data/networks increase.
A true zero trust platform that keeps all of the doors closed or "instances/vm" offline until (the milliseconds) they're needed is the security symphony we might see on the horizon.
Data silos and walled gardens may never go out of style, they'll just take on new acronyms.
A centralised graph representation of the security/functionality properties would be a better way to represent this information, so it can catch adding interfaces which should be forbidden. Also able to be configuration managed as sets of microservices.
If you have a connectivity graph it would be good to do taint analysis to see how far bad information can propagate.
Also didn't see any mention of prior art like https://cloud.google.com/beyondcorp/.
Thanks for the great writeup!
Is there a link about how much Go does Monzo they use?
- it's unambiguous
- with a proper XSD schema you can leverage validation already during parsing, which means you don't even have to boot your application at all - simply load your configuration in any XML parser and you'll automatically catch errors.
The solution is to always quote strings.
[1] https://medium.com/@lefloh/lessons-learned-about-yaml-and-no...