Our journey from a Python monolith to a managed platform
dropbox.tech
dropbox.tech
Does this imply that the atlas team gets into the weeds of understanding the business and business logic behind these endpoints to know the scalability and throughput needs? Is the autoscaler really good enough to handle this? If it's transparent to the product team, are they aware of their usage (potentially unexpected)? I imagine the atlas team would have to be very large with these sorts of responsibilities.
From a product team perspective I imagine they are still responsible for database configuration and tuning? Has the daily auto-deployment led to unexpected breaks? Who is responsible for rollbacks? And is the product team responsible and capable of hotfixes?
Maybe a more broad question which all of my questions above speak to: how are the roles and responsibilities set up between the atlas team and the product engineering team that owns the code, and how has the transition to that system been?
"We evaluated using off-the-shelf solutions to run the platform. But in order to de-risk our migration and ensure low engineering costs, it made sense for us to continue hosting services on the same deployment orchestration platform used by the rest of Dropbox."
It sounds like they acknowledge they're reinventing a lot of stuff but for now are sticking to their internal platform. Perhaps Atlas is a half-step then to get teams used to owning and running their code as isolated services. But everything I read that they built in Atlas--isolated orchestrated services, gRPC load balancing, canary deployments, horizontal scaling, etc.--are bog standard features of Kubernetes today. I'd be very leery of maintaining a bespoke Kubernetes-like platform in 2021 and beyond--in some ways it seems like it's just shifting the monolith technical debt into an internal Atlas platform team's technical debt. What's the plan to get rid of that debt for good I wonder?
This hurdle shows there's already some cracks in the idea of long-term Atlas too:
"While splitting up Metaserver had wins in production, it was infeasible to spin up 200+ Python processes in our integration testing framework. We decided to merge the processes back into a monolith for local development and testing purposes. We also built heavy integration with our Bazel rules, so that the merging happens behind the scene and developers can reference Atlasservlets as regular services."
If I read that right does it really mean the first time a developer's code is run like it will run in production is when it goes out to canary deployment? I.e. integration tests are done in a local monolith instead of setting up a mini-prod cluster. That seems a bit nerve-racking as a dev to have no way to really test the service until bits are hitting user requests. In the k8s world a ton of work has been put into tooling and processes to make setting up local clusters easily. It's a shame to not have something similar for Atlas.
> in some ways it seems like it's just shifting the monolith technical debt into an internal Atlas platform team's technical debt.
This is a key insight into the monolith problem. How does a monolith become poor and unmaintainable? A monolith becomes poor quality and unmaintainable when there is no entity enforcing architectural simplicity. It becomes unmaintainable when there is no team focusing solely on how the monolith functions. It becomes unmaintainable if there is no entity capable of saying "no" to a product engineer. A monolith in a company with weak leadership is a tragedy of the commons where everyone takes from the commons by adding complexity and there is no governing entity to ensure that the commons remains viable.
The exact statement you made is the key strength of this approach. Where there was a vacuum of responsibility before (monolith technical debt), a team has been created with direct responsibility and authority creating a governing force over that technical debt/overall complexity and therefore an entity directly responsible for improving it. This is a key first step. Atlas appears to be a compromise solution rather than an ideal end state.
Having worked in a company where no single team "owned" the monolith, the term "communally owned" tended to come up.
It was generally understood within the platform teams that if everyone owns it then in reality, no one owns it :)
The greater problem is that the ways to fail monolith specific work vastly outnumbers the way to succeed. Furthermore, the primary work involved is months to years worth of horrible unattractive refactoring. Then this is compounded by it being very hard to measure success. So you run into a situation that makes principal engineers run or look for the door. Rather than working on an actually hard problem, they would rather pad their resumes with increasingly complex novel work, further detracting from the commons. So not only is it an un-owned common, but senior engineer alignment is directly against work involving the root problem because no one in their right mind would be responsible or accountable for something with so few avenues to success.
Not only is senior engineering alignment individually against being directly responsible for the monolith, but managerial leadership is as well. You have infrastructure with infrastructure concerns, and product worried about shipping product, and then you have to ask where this monolith responsible team is. Pushing code, monitoring, capacity management, oncall, etc. are all traditionally under the infra umbrella. Business logic is traditionally under the product umbrella. So who takes responsibility for the monolith, the chief repository of business logic? A monolith is a marriage between the two entities. Each individual product team doesn't want to do monolith work, because it makes it harder to ship the features they promised. Infra doesn't want to do monolith work because it is business logic code that needs to be refactored. So no infra team and no product team will directly feel responsible. Senior managerial leadership is at the mercy of senior engineering leadership, and therefore also cannot claim responsibility for the monolith. If you travel up the management hierarchy, that leaves roughly 4 people capable of doing anything about this most critical problem. The CEO, who has ultimate responsibility for all problems, the VP of infra, who could choose to claim responsibility for this area, the VP of product, who could choose to claim responsibility for this area, or the CTO who is the most directly responsible person for solving this problem as the de facto mediator of product and infra concerns and responsibilities. The answer is probably inventing a 3rd VP responsible for the area between product and infra and giving them proper authority.
This enforcement of communal contract is toil. The only way to enforce it without toil is via spending engineering time creating systems. Maybe they are dependency injection frameworks, maybe they are servlets, maybe it's refactoring, maybe it's dependency elimination, maybe it's migration, but it is all fundamentally planned engineering work that will come at the cost of other potential work and therefore be at the mercy of management's priorities. Furthermore this kind of work must be opinionated and have the backing of internal authority, because people will absolutely violate the communal contract to get work out faster, especially senior engineers with lots of clout.
From a managerial perspective, you plan work on some cadence and expect work to get done. The engineers who are most effective at doing what they said they would do are the ones who are most going to please you, and therefore the ones who are going to get promoted. All toil comes directly at the cost of this. In order for this work to be internally incentivized it must be done as part of the companies overall planning and reward system.
Remember that this communal ownership is over a commons (overall complexity), and just like cows over grazing a field and running out of food, engineers over produce complexity and run out of time (time gets spent pushing code, dealing with bugs, running tests, refactoring code, etc.). When individual incentives are sour, you must have a governing body regulate the system. Each cow is individually incentivized to eat their fill, and each engineer is individually incentivized to ship product asap, regardless of code quality.
Most damning of all to communal enforcement is that as the number of engineers scales up, any non-explicit contract will be unenforceable as the number of communal "enforcers" with reasonable/relevant scope disappears entirely.
"Metaserver was stuck on a deprecated legacy framework that unsurprisingly had poor performance and caused maintenance headaches due to esoteric bugs. For example, the legacy framework only supports HTTP/1.0 while modern libraries have moved to HTTP/1.1 as the minimum version."
Dropbox has been around for a lot of years, and raised a lot of cash; was it only recently that they could pay down this technical debt? Were they really so busy in other areas that this was allowed to fester?
The legacy framework was Pylons, which eventually evolved into Pyramid.
The tldr is there were hundreds of unowned endpoints that, yes, were allowed to fester. They eventually got ownership on all endpoints, so you had somebody to exert pressure on to make things happen.
I am know very little about Python, but does this mean that Python has no way to encapsulate code at a level larger than a class? Something like a package or a module. It does not seem like it should be necessary to break a system into separate services just to get encapsulation at a module of subsystem level.
import foo
foo.auth("admin", "badpass") returns False
def new_auth(user, pass):
. . . .return True
foo.auth = new_auth
foo.auth("admin", "badpass") now returns True
Python has packages, which contain modules. This doesn't seem to be a reference to the lack of encapsulation, in any case.
Though it seems to be projecting a social problem (absence of a policy involving active decisions on when particular code can be used by endpoints) onto architecture.
Granted, you shouldn't do very much of that except in some extremely specialized and careful circumstances, but in a language that makes it really easy, you have to be extremely disciplined and often delivery velocity takes precedence over careful planning of what really constitutes a public API.
Practically speaking, putting separate services in literally separate process spaces where the only way they can communicate is via message passing is the only way to really enforce encapsulation.
The language designers definitely took the path of we're just going to assume you know what you're doing and give you absolutely every last foot of rope you could possibly want to hang yourself with.
It does make things fast and easy. See all of the complaints here lately about having to rewrite significant parts of an application to become async as soon as any part of it becomes async? Not a problem in Python. There's a library to just automatically rewrite all code as it is being imported, at the cost that the code you're actually running is not the code you see in your repos.
def make_get_private(var):
. . . .private_var = var
. . . .def get_private():
. . . . . . . .return private_var
. . . .return get_private
get_private = make_get_private("spam")
You can't readily modify private_var after it is set in this case, you could also make and return a set_private function as well. I imagine you could do a whole module like this and then have it create it's functions at runtime to produce this "private" namespace. It would be a mess though.
So... about this headline. I read this aloud to a friend at a cafe. We laughed. It makes perfect sense to us. We know what Python is. We know what a monolith means in this context.
To my other friends it was the funniest / silliest / nonsensical thing they'd heard for awhile.
IT is weird.
(ps I know no one will see this comment but I'll leave it here. Because.)
https://dropbox.tech/infrastructure/rewriting-the-heart-of-o...
Then I don’t understand the delay to shipping an Apple Silicon build. Right now we still have to used Rosetta... it’s the only such piece of software I have that does.
If you have a specific version per subproject, how do you track that in the repo? Different tag schemes for different subprojects? I have used that in a small-ish monorepo and I didn't like it specially.
If we were doing continuous delivery, too, I could see there not being much value in messing with independent versioning, semver, whatever. Just make today's date the universal version numbering system for all modules and move along.
For example, you are updating service B to call new endpoint on service A. First you need to make service A endpoint available, and then make service B call service A.
Just because everything exists in the same repo does not mean it all gets shoved out at once. The downside is that you can't just read the code and assume the running service is doing that, unless it's embedded in your build. Processes like automated updates and a forced update cadence (no running binaries over X days old) with proper canary/vetting before a full release allow a large org to still manage this complexity.