This is a strategy I’ve used across many projects in several languages, from big Rails apps to the Go monolith we deploy at incident.io today.
It’s just one way you can make a monolith much more robust without moving into separate services, which helps you keep the benefits of a monolithic application for longer.
Hope people find it interesting.
How do you coordinate the efforts of 200 engineers on a single repo/code base?
Do engineers frequently get into long/drawn-out merge sessions as common code may be modified by a number of engineers who are all trying to merge around the same time? This is actually one of the reasons I really like GOLANG: "A little copying is better than a little dependency."
- Merging was automated (a robot tried to run tests against fresh master and merge only if green).
- Deploy was fully automated and limited to working hours.
- We added tests for problematic parts. For example static analysis for database migrations to prevent only safe actions in an automated fashion.
However, if something goes wrong in some component, you have to revert and stop the deploys for everyone which sucks. I'd say around 8 - 10 deploys per day, it makes sense to start splitting the components or at least not adding new teams to the same monolith.
Surprisingly, coordination isn’t that difficult. Humans are pretty good at talking to each other.
> But having spent half a decade stewarding a Ruby monolith from 20 to 200 engineers and watched its modest 10GB Postgres database grow beyond 5TB, there’s definitely a point where the pain outweighs the benefits.
Woah!
I for one would also love to hear your insights on scaling personnel from 20 to 200.
We’re in a similar boat / anticipated growth phase (at 20 odd engineers), and whilst there’s a lot of content on this topic, I’d appreciate your practical “from the coal face” take (much like what you’ve done with this monolith article).
Perhaps a follow up article? ^_^
The experience I reference was from my time at GoCardless, where I was a Principal SRE leading the team who were on-call for the big Rails monolith.
I’ll put the topic of “what does 10x eng team scaling feel like” on my todo list for posts, but if you’re interested there’s a bunch of articles on my personal blog from this time.
One that might be interesting is “Growing into Platform Engineering” about how the SRE function evolved:
https://blog.lawrencejones.dev/growing-into-platform-enginee...
The reasons we went with one binary are:
- We can ensure common runtime setup by only having one codepath that runs the initialisation. It's nice that every deployment definitely instantiates the Prometheus server and initialises secrets from Google Secret Manager at the same point and in the same way.
- Compiling three targets adds time to our build process. Even when Go will cache much of the compilation, linking separate binaries is still another ~30s and ~300MB of binary to our resulting docker image.
- It's very convenient in development to run all deployments inside of a single binary for hot-reloading purposes. This is how we develop the app locally.
I don't think it makes much difference which you choose in terms of the operational elements. But for the reasons above, there are clear advantages to the single binary for our use cases.