In Defense of Simple Architectures (2022)
danluu.com
danluu.com
Theoretically If you have a monolith that can be scaled horizontally there isn't any difference between having 10 replicas of your monolith and having 5 replicas of two microservices with the same codebase. UNLESS you are trying to underscale part of your functionality. You can't underscale part of your app with a monolith. Your pipe has to be big enough for all of it. Generally speaking though if you are talking about 10 replicas of something there's very little money to be saved anywhere.
Even then though the cost savings only start at large scales. You need to have a minimum of 3 replicas for resiliency. If those 3 replicas are too big for your scale then you are just wasting money.
The place where I see any real world benefit for most companies is just engineering coordination. With a single repo for a monolith I can make 1 team own that repo and tell them it's their responsibility to keep it clean. In a shared monolith however 0 people own it because everyone owns it and the repo becomes a disaster faster than you can say "we need enterprise caching".
On your note about resiliency and scale - its always a waste of money until shit hits the fan. Then you really pay for it.
Added bits of "resiliency" often add brand new, unexplored failure points that are just ticking time bombs waiting to bring the entire system down.
I can tell you 100% that eventually a disk will fail. I can tell you 100% that eventually the power will go out. I can tell you 100% that even if you have a computer with redundant power supplies each connected to separate grids, eventually both power supplies will fail at the same time - it just will happen a lot less often than if you have a regular computer not on any redundant/backup power. I can tell you that network cables do break from time to time. I can tell you that buildings are vulnerable to earthquakes, fires, floods, tornadoes and other such disasters). I can tell you that software is not perfect and eventually crashes. I can tell you that upgrades are hard if any protocol changed. I can tell you there is a long list of other known disasters that I didn't list, but a little research will discover.
I could look up the odds of the above. In turn this allows calculating the costs of each mitigation against the likely cost of not mitigating it - but this is only statistical you may decide something statistically cannot happen and it does anyway.
What I cannot tell you is how much you should mitigate. There is a cost to each mitigation that need to be compared to the value.
The theoretical benefit may not be what most teams are going to experience. Usually the fact that microservices are seen as a solution to a problem that could more easily be solved in other much simpler ways, is a pretty good indication that any theoretical benefits are going to be lost through other poor decision making.
You can certainly misapply the architecture, but you can also apply it well. It's unsurprising that most people make bad choices in a difficult domain.
But for some reason it seems that few people are working on making our programming languages and frameworks fault tolerant.
I've never really understood why you couldn't just break up your monolith into modules. So like if there's a "payments" section, why isn't that API stabilized? I think all the potential pitfalls (coupling, no commitment to compatibility) are there for monoliths and microservices, the difference is in the processes.
For example, microservices export some kind of API over REST/GraphQL/gRPC which they can have SDKs for, they can version them, etc. Why can't you just define interfaces to modules within your monolith? You can generate API docs, you can version interfaces, you can make completely new versions, etc.
I just feel like this would be a huge improvement:
- It's so much more engineering work to build the service handler scaffolding (validation, serialization/deserialization, defining errors)
- You avoid the runtime overhead of serialiation/deserialization and network latency
- You don't need to build SDKs/generate protobufs/generate clients/etc.
- You never have the problem of "is anyone using this service?" because you can use code coverage tools
- Deployment is much, much simpler
- You never have the problem of "we have to support this old--sometimes broken--functionality because this old service we can't modify depends on it". This is a really undersold point: maybe it's true that microservice architectures let engineers build things without regard for other teams, but they can't remove things without regard for other teams, and this dynamic is like a no limit credit card for tech debt. Do you keep that service around as it slowly accretes more and more code it can't delete? Do you fork a new service w/o the legacy code and watch your fleet of microservices grow ever larger?
- You never have the problem of "how do we update the version of Node on 50 microservices?"
And instead you have the problem of "how do we update the version of Node on our 10 million LOC codebase?" Which is, in my experience, an order of magnitude harder.
Ease of upgrading the underlying platform versions of Node, Python, Java, etc is one of the biggest benefits of smaller, independent services.
CI/CD is also an incentive to keep domain-level scope changes small (scope creep tends to be a problem in software development) in order to minimize disruptions to the pipeline.
These are all somewhat different problems than upgrading the platform you're running, which the test suite itself should cover.
Yes, change failure rate is also measured, and that's why good test suites matter, but if you think frequency and scope of change don't matter for successful projects, you haven't looked at the data.
That means frequently updating your dependencies against a small code base is much more useful (and painless) than occasional boil-the-ocean updates.
(As always, excepting small-ish teams, because direct communication paths to everybody on the team can mitigate a lot of problems that are painful at scale)
I think if you get to that scale everything is pretty hard. You'll have a hard time convincing me that it's any easier/harder to upgrade Node on 80 125K LOC microservices than a 10M LOC monolith. Both of those things feel like a big bag of barf.
I think part of it is that many just don't know how.
Web developers deal with HTTP and APIs all the time, they understand this. But I suspect that a lot of people don't really understand (or want to understand) build systems, compilers, etc. deeply. "I just want to press the green button so that it runs".
You can! We used to do this! Some of us still do this!
It is, however, much more difficult. Not difficult technically, but difficult because it requires discipline. The organisations I’ve worked at that have achieved this always had some form of dictator who could enforce the separation.
Look at the work done by John Lakos (and various books), to see how well this can work. Bloomberg did it, so can you!
Creating a network partition makes your system a distributed system. There are times you need this, but the tradeoff is at least an order of magnitude increase in complexity. These days we have a lot of tooling to help manage this complexity, but it’s still there. The combination of possible failure states is exponential.
Having said all this, the micro service architecture does have the advantage of being an easy way to enforce modularity and does not require the strict discipline required in a monolith. For some companies, this might be the better tradeoff.
Like a lead developer or architect? Gasp!
I wonder if the microservices fad is so that there can be many captains on a ship. Of course, then you need some form of dictator to oversee the higher level architecture and inter-service whatnots... like an admiral.
If you just follow the SOLID principles, you're already 90% there. If your team doesn't have the knowledge (it's not just "discipline", because every proper developer should know that they will make it harder for everyone including themselves if they don't follow proper architecture) to write structured code, letting them loose on a distributed system will make things much, much worse.
You can sort of get there if you have a strong central team working on monolith tooling that enforces module seperation, lints illegal coupling, manages sophisticated multi deployments per use, allows team based resource allocation and tracking, has per-module performance regression prevention, etc. They end up having many of the organizational problems of a central DBA team, but its possible. Even then, I am not aware of many(any?) monoliths in this situation that have scaled beyond 500+ engineers that people are actually happy with the situation they've ended up in.
How so? If your microservices are in a monorepo, one dev can spread joy and disaster across the whole ecosystem. On the other hand, if your monolith is broken into libraries, each one in its own repo, a developer can only influence their part of the larger solution. Arguably, system modularity has little to do with the architecture, and much to do with access controls on the repositories and pipelines.
Monoliths tend to be in large monolithic repos. Microservices tend to get their own repo. Microservices force an API layer (defined module interface) due to imposing a network boundary. Library boundaries do not, and can generally be subverted.
I agree that modularity has nothing to do with the architecture, intrinsically, simply that people are pushed towards modularity when using microservices.
In my experience, microservices require more discipline than monoliths. If you do a microservice architecture without discipline you end up with the "distributed monolith" pattern and now you have the worst of both worlds.
Before that, people need to know it's even an option.
Years ago when I showed a dev who had just switched teams how to do this with a feature they were partway through implementing (their original version had it threading through the rest of the codebase) it was like one of those "mind blown" images. He had never even considered this as a possibility before.
And that is why developers have so much trouble getting it right. They can't without having the organisational fundamentals in place. It is simply not possible.
The architectural constraints of microservices will show the organisational weaknesses in a much higher rate because of the pressure it puts on having the organisation be very strict about ownership, communication and autonomy.
The takes a higher level of maturity as an organisation to enable the benefits of microservies, which is also why most organisations shouldn't even try.
Stop all the technical nonsense because it won't solve the root cause of the matter. It's the organisation. Not the technology
But yeah, they also tend to explode complexity
For instance, you might not want to require all develops have a C tool chain installed with certain libraries for a tiny bit of performance optimized code that almost never gets updated.
For development purposes you could build and distribute binary artifacts just like you would for any other library. Developers who don't touch native code can just fetch the pre-built binaries corresponding to current commit hash (e.g. from CI artifacts).
That's a pretty common approach. Service oriented architecture isn't necessarily "development" purposes, usually more generic "business" purposes but organization layout and development team structure are factors.
From an operational standpoint, it's valuable to limit the scope and potential impact code deploys have--strong boundaries like the network offers can help there. In that regard, you're serving a "development" purpose of lowering the risk of shipping new features.
My overriding memory of trying to do literally anything with Ruby was having Nokogiri fail to install because of some random missing C dependency on basically every machine...
Nodejs, as far as I know, ships entire platform specific tool chains as dependencies that are downloaded with the dependency management tool. Python goes a different direction and has wheels which are statically linked binaries built with a special build tool chain then hosted on the public package repo ready to use.
I don't think anything in Ruby has changed but I haven't used it in a few years.
I'm not saying microservices are the right answer. They are a useful tool that sometimes you need and sometimes you don't. You should have someone on your team with enough experience to get the decisions close enough up front. This isn't anything new.
It doesn’t take a microservice to allow rewriting hot paths in a lower level language. Pretty much everything has a C-based FFI. This is what makes Python useable for ML—the libraries are written in C/C++.
Who thinks that more i/o and more frequent cold starts are more performant?
Where I see micoservices as very useful is for elasticity and modularization. It's probably slower than a monolith at your scale but you don't want to fallover when loads start increasing and you need to scale horizontally. Microservices with autoscaling can make that very useful.
But of course, updating services can be a nightmare. It's a game of tradeoffs.
Yeah they were a way to handle databases that couldn't scale horizontally. You could move business logic out of the database/SQL and into Java/Python/TypeScript app servers you could spin up. Now that we have databases like BigQuery and CockroachDB we don't need to do this anymore.
As if you have say 100x more customers, will you divide your application in 100x more services?
As others said, microservices scale manpower, not performance. It only comes sometimes as a bonus.
But I think you are pointing out one of the reasons most places don't get benefits from microservices. If their culture doesn't let them do as you describe with a monolith, the overhead of microservices just bring in additional complexities to an already dysfunctional team/organization.
I assure you 98% of the companies have simpler and shallower code base than that of a modern A class browser.
Microservices was a wrong turn in our industry for 98% of the use cases.
This made me chuckle. :) It's HarfBuzz. Yours sounds like a goblin.
But yes, you're right.
i've always felt the org chart defines the microservice architecture. It's a way to keep teams out of each other's hair. When you have the same dev working on more than one service then that's an indication you're headed for trouble.
Most companies can run on a VPS.
Similarly, if you have any critical endpoints that are read-only, splitting those out from the main service where writes occur can improve the reliability of those endpoints.
What do you mean by "pipe" here? It's easier to share CPU and network bandwidth across monolith threads than it is across microservice instances. (In fact, that is the entire premise of virtualization - a VM host is basically a way to turn lots of disparate services into a monolith.)
* I give two teams one repo each and tell each one: "This is your repo, keep it clean."
-or-
* I give two teams one folder each in a repo and tell each one: "This is your folder, keep it clean."
If you've got a repo or folder (or anything else) that no one is responsible for, that's a management problem, and micro services won't solve it.
Repos (and folders) don't really have anything to do with organizing a complex system of software -- they are just containers whose logical organization should follow from the logical organization of the system.
Microservices back you into position where when the code of one team to calls the component of another team, it has to be high-latency, fault-prone, low-granularity, config-heavy. Some stuff falls the way anyway, so no problem in those case, but why burn that limitation in to your software architecture from the start? Just so you don't have to assign teams responsibility for portions of a repo?
you lose some CPU/RAM/IO efficiency, but you gain autonomy and resilience, and faster time to resolution.
https://blog.nelhage.com/post/efficiency-vs-resiliency/
e.g. at Grafana we're working through some of these decoupling-to-microservices challenges, because the SREs that deploy to our infra should not need to deal with investigating and rolling back a whole monolith due to some regression introduced by some specific team to a core datasource plugin, or frontend plugin/panel. the pain at scale is very real, and engineering hours are by far more expensive than the small efficiency loss you get by over-provisioning the metal a bit.
But I will note that almost all the benefits listed aren't specific to micro services.
(Also, to me, it's flawed that a dependency can really push to production without coordination. At the least, dependent components should decide when to "take" a new version. There are lot of ways to manage that -- versioned endpoints, e.g., so that dependents can take the new when they are ready. But it's extra complexity that's tricky in practice. e.g., you really ought to semver the endpoints, and keep versions as long as there could be any dependents... but how many is that, and how do you know? From what I've seen, people don't do that and instead dependencies push new versions, ready or not, and just deal with the problems as breakages in production, which is pretty terrible.)
Microservices supposed to be so small that they are much smaller than a team's responsibility; so I don't think that is a factor.
The bottleneck in a software engineering company is always leadership and social organisation (think Google - they tried to muscle in on any number of alternative billion dollar businesses and generally failed despite having excellent developers who assembled the required parts rapidly). Microservices won't save you from a a manger who doesn't think code ownership is important. Monoliths won't prevent such a manger from making people responsible for parts of the system.
I've worked on monoliths with multiple teams involved. It seems natural that large internal modules start to develop and each team has their own areas. A little bit like an amoeba readying to split, but never actually going through with it.
Not necessarily. I owned one of Amazon’s most traffic-heavy services (3 million TPS). It was a monolith with roughly 20 APIs, all extremely high volume. To keep the monolith simple and be able to scale it up or down independently, the thousands of EC2 instances running it were separated in fleets, which serving a specific use case. Then we could for example scale the website fleet while keeping the payment fleet stable. The only catch is that teams needed to be allowed to call only the APIs representing the use case of that fleet (no calling payment-related APIs on the website fleet). Given the low number of APIs and basic ACL control, it was not a challenge.
Queue my favourite talk about microservices: David Schmitz - 10 Tips for failing badly at Microservices [1]
This guy has amazing delivery -- so dry and funny. He spends 45 minutes talking about all of his microservices mistakes!
I do think things have gotten a little silly in many places with too much “building like we’re Netflix” because often your microservices can easily be what is essentially a bunch of containerised monoliths.
I think the main issue is that your IT architecture has or should have) very little to do with tech and everything to do with your company culture and business processes. Sometimes you have a very homogeneous focus maybe even on a single product, in which case microservices only begin to matter when you’re Netflix. Many times your business will consist of tens-thousands of teams with very different focuses and needs, and in these cases you should just never do monoliths unless you want to end up with a technical debt that will hinder your business from performing well down the line.
However I doubt Netflix has actually documented their architecture in enough detail that you could use it. Even if you hire Netflix architects they may not themselves know some important parts (they will of course know the parts they worked on)
By the way - Netflix started as a monolith, and so did most other big services that are still around.
The rest faded away, crushed by the weight of complexity, trying to be "like Netflix".
> Monoliths aren’t very useful in many organisations where you need to build and connect 300+ systems
A Google team published a paper on this last year: https://dl.acm.org/doi/10.1145/3593856.3595909
> When writing a distributed application, conventional wisdom says to split your application into separate services that can be rolled out independently. This approach is well-intentioned, but a microservices-based architecture like this often backfires, introducing challenges that counteract the benefits the architecture tries to achieve. Fundamentally, this is because microservices conflate logical boundaries (how code is written) with physical boundaries (how code is deployed). In this paper, we propose a different programming methodology that decouples the two in order to solve these challenges. With our approach, developers write their applications as logical monoliths, offload the decisions of how to distribute and run applications to an automated runtime, and deploy applications atomically. Our prototype implementation reduces application latency by up to 15× and reduces cost by up to 9× compared to the status quo.
But that’s not what happens on enterprise organisations. 90% of those are bought “finished” products, which then aren’t actually finished and you can be most certain that almost none of them are capable of sharing any sort of data without help.
Hell, sometimes you’ll even have 3 of the same system. You may think it’s silly, but it is what it is in non-tech enterprise where the IT department is viewed as a cost-center similar to HR but without the charisma and the fact that most managers think we do magic.
Over a couple of decades I’ve never seen an organisation that wasn’t like this unless it was exclusively focused on doing software development, and even in a couple of those it’s the same old story because they only build what they sell and not their internal systems.
One of the things I’m paid well to do is help transitions startups from their messy monoliths into something they can actually maintain. Often with extensive use of the cheaper external developers since the IT department is pure cost (yes it’s silly) and you just can’t do that unless you isolate software to specific teams and then set up a solid architecture for how data flows between systems. Not because you theoretically can’t, but because the teams you work with often barely know their own business processes. I currently know more about specific parts of EU energy tariffs than the dedicated financial team of ten people who work with nothing else, because I re-designed some of the tools they use and because they have absolutely no process documentation and a high (I’m not sure what it’s called in English, but they change employees all the time). Which is in all regards stupid, but it’s also the reality of sooo many places. Like, the company recently fired the only person who knows how HubSpot works for the organisation during down sizing… that’s the world you have to design systems for, and if you want it to have even a fraction of a chance to actually work for them, you need to build things as small and isolated as possible going all in on team topologies even if the business doesn’t necessarily understand what that is. Because if you don’t, you end up with just one person who knows how the HubSpot integrations and processes work.
It’s typically the same with monoliths, they don’t have to be complicated messes that nobody knows how work… in theory… but then they are build and maintained by a range of variously skilled people over 5 years and suddenly you have teams who hook directly into the massive mess of a DB with their Excel sheets. And what not.
To help you out, the word is attrition or turnover. Turnover would be more appropriate if the roles are refilled, attrition if the roles are never replaced.
The most frequent problem that I see is that people design Microservices that are "small as possible" instead of "Small enough than a small team can own them". For example, I have seen teams of 5-7 developers owning 20 microservices which is crazy IMHO. I blame the pre-fix micro which is highly miss leading. Now, if you merge together these 20 tiny microservices doesn't make monolith. A monolith would be a service or application that is owned by multiple/many teams.
The other important aspect is that microservices should be loosely coupled and what I see is highly coupled microservices. How do you know you have highly coupled microservices? I have seen a few things.
- They share business logic and developers end up creating libraries with business logic which is very problematic.
- They share access to the tables.
- Changes in one often require changes in other microservices.
- Integration and unit tests are not enough. You need E2E tests, but these are hard to write and brittle.
Here's a concrete example, stolen from JD Long's great NormConf talk, "I'd have written a shorter solution but I didn't have the time", which is itself drawing from "People systematically overlook subtractive changes" by Adams, et al., 2021:
Give people a Lego construction consisting if a large, sturdy plinth with single small spire supporting a platform on top. The challenge is to get it to reliably support a brick. As is, it will collapse because that little spire is so flimsy. Ask participants to modify the structure so that it will support the brick. Incentivize simpler solutions by saying that each additional brick costs $0.50.
Given just this information, people universally try to fix the problem by adding a bunch more bricks to overbuild the structure, and they spend a lot of energy trying to come up with clever ways to reduce the number of bricks they add. But then the research team makes one small tweak to the instructions: they explicitly point out that removing bricks doesn't cost anything. With that nudge, people are much more likely to hit on the best solution.
Additive changes have the advantage of having a corresponding additive feature to test - you don't need to touch or even understand any of the existing complexity, just glue it on top.
So the cost structure is likely inverted from the study you describe. Additive changes are (locally) cheap. Subtractive changes are potentially expensive.
I don't know whether the Lisps have a similar "fun to refactor" quality to them, since while they are in the functional camp, they're totally different and more flexible beasts architecturally.
Fortunately, Lisp codebases IME tend to be smaller and use small, unlikely-to-need-to-be-modified macros and functions, so large refactors may be less common.
Yes you might get to change the goal in business software development, but you also know that the goal is 'working solutions' not 'build up this existing structure to do X'.
But that is hard to do, and it takes experience. And one thing that does not describe the IT industry well is "experience": don't most software developers have less than 5 years experience? You can read all the books you want and pass all the certifications you can like a modern Agile manager, but at the end of the day you will have the same problem they have: experience takes time.
Inexperienced engineers throw sexy tech at everything, inexperienced manager throw bullshit Agile metrics at everything. Same fight.
You are saying that some people with 20 years of experience are worse architects than others.
Or as Albert Einstein phrased it: “Everything should be made as simple as possible, but not simpler!”
Also, the area of experience matters. I myself for instance have become pretty good at designing systems and infrastructure both from scratch up to corporate level. However, that will not help me too much when it comes to mega-corp level (think: FAANG) and other things such as designing public opensource libraries. (libraries and systems/applications require very different tradeoffs).
You're looking for a lifestyle business.
They are great. You will get time in the garden, that's awesome.
Vacation someplace with no phones? Out of the question. Weekend in a 3rd world country with character and food and sketchy internet. Not gonna happen.
You want to optimize for free time in the garden by all means you can do it, but you loose out in other places, you pick up other work (taxes, systems etc).
Edit: Down vote away, I live this life now, my tomatoes are delicious and I make yogurt too!
Still never work weekends (no phone, no email, no slack), spend time in my garden and just came back from a no-phone vacation.
Salary is above the going rate where I live - the work is remote and salary is based on company headquarters. Taxes are in line with the country I live in.
Not really seeing any downsides here, and as expected morale is quite good overall at work... But finding this company was extremely lucky/difficult.
Sure I can write code for 14 hours a day, but it will be bad code, sometimes even negative productivity as I introduce so many bugs.
Software for youth sports, photography, asset tracking, vendor tracking, niche issues in CC Processing, facets of insurance and billing (coding).
Niche businesses happen all over the place, and finding one (for me) was a lot of trial and error, that niche business pays my "bills" and I do consulting work (sporadic, interesting and high value) to round it out (and keep me on my game).
Dont think of it as a business right away. You're going to "play", you want to build them quickly, you want to host them cheaply, you want to toy with selling them. Your managing disappointment and failure, your learning lessons and keeping it FUN. The moment you start dreaming that it's going to "make it big" is the moment you have to reel yourself back to reality. If you can say "what did I learn" and have a list of things you got from it then it was a success. At some point you just find one that clicks and it grows.
The company appears dogmatically locked into their idling python code, their SQL making unpredictable commits, and their SQL framework making it difficult to make schema migrations.
This is a financial services company describing data-integrity bugs in their production platforms.
I would argue the opposite - the more distributed you make the system, the more subtle consistency bugs tend to creep in.
Conversely, I find myself constrained by C#'s typesystem not having features now widely popular from Rust and Typescript and available in modern Python: const generics / literals, protocols/structural interfaces and matching over algebraic datatypes, to name a few examples.
Nor "matching over algebraic datatypes" made any sense - sure sounds slightly below average fancy but comes down to just using C# pattern matching effectively, which the author may have never done. But then it struck me "structural interfaces sure sounds familiar", I go check the pinned repositories in the github profile and it all made sense immediately - the pinned ones are written in Go, which explains the author's woes with confidently navigating something more sophisticated.
The question I responded to said: "Better static analysis, for one."
I asked: "What is something you can express in the C# type system that MyPy cannot statically check?"
Can you provide some examples where C#'s type system can express something that modern well-typed Python cannot type-check?
Can you help guide me where I'm misunderstanding the type system aspect of const generics that's missing in Python? What I meant was that in Python I can say something like this:
import typing as t
class FooEater[NUM_FOOS: int]:
def eat_foo(self, num_foos: NUM_FOOS):
print("nomnom")
one_foo = FooEater[t.Literal[1]]()
one_foo.eat_foo(2) # type error, NUM_FOOs for one_foo only allows literal '1'
I admit to not being a type system specialist, but at least as I understand Rust's "const generics" the type system feature is: "Things can be generic over constant values". Isn't that the same as what Python supports?
And if you're doing that, python has the Decimal class for the same thing.
Here's what I see them as providing:
- Simple async, no wasted CPU on I/O
- Strong typing defaults
- Reliable static analysis
Half of the "pros" he lists for using GraphQL are just provided out of the box using ASP.NET Core with NSwag to document the endpoints. If they want to keep the client-side composability, create some OData endpoints and you've got it.
> Self-documentation of exact return type
> Code generation of exact return type leads to safer clients
> Our various apps (user app, support app, Wave agent app, etc.) can mostly share one API, reducing complexity
> Composable query language allows clients to fetch exactly the data they need in a single packet roundtrip without needing to build a large number of special-purpose endpoints
Bias flag: I'm primarily a C#/.NET developer. I came up on python and ruby but have had more success in my region getting paid to work on .NET code.
On the actual criticism you’re raising: In what concrete way would moving to micro services help with reducing data integrity bugs? In my experience I’d expect the exact opposite - micro services systems generally being harder to test and generally having more possible interleavings and partial failure paths.
What I can offer is a personal anecdote about what an amazing guy he is. I had been a fan of his blog for a while, and at one point I decided to just email him and offer to fly to wherever he was because I wanted to hear his thoughts on a number of topics.
This was in maybe 2016, but even then I didn't expect someone who must get a zillion such emails to even reply, let alone invite me up to Seattle at my leisure! I think I had the flight booked on a Wednesday for a departure on Friday, for one night's stay, and Dan was profoundly generous with his time, we stayed up late into the night chatting and I learned a great deal, especially about the boundary between software and hardware (a topic on which he is a first-order expert with an uncommon gift for exposition).
I had the great fortune of spending some real time with Dan not just once but twice! When I went to NYC to get involved with what is now the Reels ML group, he happened to also be in NYC, and I had the singular pleasure to speak with him at length on a number of occasions: each illuminating and more fun than you can have without a jet ski.
Dan is a singularly rigorous thinker with the dry and pithy wit of a world-class comedian and a heart of gold, truly generous with his expertise and insight. I'm blessed to have met and worked with a number of world-class hackers, but few, few if any are such a joy to learn from.
But.
I've dealt with way too many teams whose shit is falling over due to synchronous IO even at laughably low volumes. Don't do that if you can avoid it.
"Subtle data-integrity bugs" are not something we should be discussing in a system of financial record. Avoiding them should have been designed in from the start.
Like? I ask because even synchronous IO let's you serve millions of requests per month on a cheap VPS.
That's enough in the b2b space to keep a company of 1000 employees in business.
Then even if you're only at millions per month you've probably got problems.
Not in my experience. You may be using metrics for B2C websites which make $1 for each 1 million hits.
B2B works a little differently: you're not putting everyone on the same box, for starters.
I did some contract maintenance for a business recently (had no tech staff of their own, had contracted out their C# based appdev to someone else decades ago and just need some small changes now), and a busy internal app serving about 8000 employees was running just fine off a 4GB RAM VPS.
Their spend is under $100/m to keep this up. No async anywhere. No performance problems either.
So, sure, what you say makes sense if your business plan is "make $1 of each 1 million visitors". If you business plan is "sell painkillers, not vitamins" you need maybe 10k paying users to pay yourself a f/time salary.
Which, in B2B, is insanely profitable. At 20 rqsts/min, for a paying customer paying you $200/m/user, those numbers are fantastic!
I can only dream of having those numbers!
(Agreed that synchronous IO can serve hundreds of requests per second with the right threading model)
I don't know the answer, but I would like to. I think it has to do with the limit on the number of processes/threads that the OS/framework can manage. Once you reach this limit, using async/await somehow allows the OS/framework to secretly use your (technically not) blocked threads to do some other work while they are (technically not) blocked.
2. OS threads can be expensive in the sense that they can block a lot of memory, to the point where you could run out of threads at some point. This is worse in some environments than in others. Apart from async, another solution for this is virtual / green threads (as in Erlang, Haskell, and much more recently, Java).
3. Some async implementations enable advanced structured concurrency patterns, such as cancellation, backpressure handling, etc.
$16/month "Short-form posts"
$256/month "Sponsor"
$2,048/month "Major Sponsor"
$16,384/month "Patron"
You certainly can't fault the guy for trying, although it does make him seem a bit crazy. And I'm aware of scenarios where raising prices doesn't have the negative effect you might expect. You never know what people might spend but I can't imagine this is a tempting proposition for anyone.
Maybe it's all a ploy to get you to spend $16/month to see how many "Sponsor", "Major", and "Patron" level subscribers there are.
Also, if you look at his other writings, he's often noted trends in how software developers are paid. And the paradox that with even small tweaks, he's sometimes saved his employers far more than he could have earned in ten lifetimes. He's a free agent now... so why not charge what his advice is actually worth?
Dan is a very unusual person, in that he tries to do what is predicted by the data he has, rather than waiting for it to become conventional wisdom.
Wave financial is only 350 people according to wikipedia - I doubt that’s 350 engineers. I know only of Google and Meta that can even operate with a massive monorepo, but I wouldn’t call their architecture “simple”. And even they do massive internal tooling investments - I mean, Google wrote their own version control system.
So I tend to think “keep it simple until you push past Dunbar’s number, then reorganize around that”. Once stable social relationships break down, managing change at this scale becomes a weird combination of incredible rigidity and absolute chaos.
You might make some stopgap utility and then a month later 15 other teams are using it. Or some other team wants to change something for their product and just submits a bunch of changes to your product with unforseen breakage. Or some “cost reduction effort” halves memory and available threads slowing down background processes.
Keeping up with all this means managing hundreds of different threads of communication happening. It’s just too much and nobody can ever ask the question “what’s changed in the last week” because it would be a novel.
This isn’t an argument for monoliths vs microservices, because I think that’s just the wrong perspective. It’s an argument to think about your social structure first, and I rarely see this discussed well. Most companies just spin up teams to make a thing and then don’t think about how these teams collaborate, and technical leadership never really questions how the architecture can supplement or block that collaboration until it’s a massive problem, at which point any change is incredibly expensive.
This also doesn't mean to not be forward thinking. You want the architecture to support growth that will more than likely happen, just keep the expectations in check.
> You want the architecture to support growth that will more than likely happen
The problem is even very experienced people can disagree about what forms of complexity are worth it up-front and what forms are not.
One might imagine that Google had a first generation MVP of a platform that hit scaling limits and then a second generation scaled infinitely forever. What actually happens is that any platform that lives long enough needs a new architecture every ~5 years (give or take), so that might mean 3-5 architectures solving mostly the same problem over the years, with all of the multi-year migration windows in between each of them.
If you're very lucky, different teams maintain the different projects in parallel, but often your team has to maintain the different projects yourselves because you're the owners and experts of the problem space. Your leadership might even actively fend off encroachment from other teams "offering" to obsolete you, even if they have a point.
Even when you know exactly where your scaling problems are today, and you already have every relevant world expert on your team, you still can't be absolutely certain what architecture will keep scaling in another 5 years. That's not only due to kinds of growth you may not anticipate from current users, it's due to new requirements entirely which have their own cost model, and new users having their own workload whether on old or new requirements.
I've eagerly learned everything I can from projects like this and I am still mentally prepared to have to replace my beautifully scaling architectures in another few years. In fact I look forward to it because it's some of the most interesting and satisfying work I ever get to do -- it's just a huge pain if it's not a drop-in replacement so you have to maintain two systems for an extended duration.
This is where we are.
I can't tell you how many times I have seen projects get done just to pad a PM or developers resume. Just because it was the lastest and greatest hot shit thing to use. No sense of if it would be better, faster, cheaper, useful.
When cloud was the hot new thing the company that I worked with launched a replatform on AWS. It gave us the ability to get through the initial scaling and sizing with ease. We left right away, because even then the costs did not make sense. Now we see folks crying about "exit fees" that were always there. That assumes that your institution even has the gall to own years of pissing away money.
Workman like functionality isnt sexy, it wont be the hot bullet point on your resume, it wont get you your next job, but it is dam effective.
Any other engineering discipline. What are common practices in IT would be negligence in other disciplines and might get your permit/license removed.
IT is the only sector where companies like Cisco or SAP can exist despite the horrible reliability of their products.
Come on, other industries have garbage companies putting out garbage products, too.
That's correct, but we have to admit that the software industry excels at this.
As in - you can setup a process in Germany, then replicate it globally with predictable accuracy. And predictability matters a lot in stable low margin businesses. Walmart can't spend a few billion on a project that may have -100% to 400% return value, when. they have the option of having a reliable 20%-30% return value.
We see it in the C-suite; we see it with engineers.
I think the travesty of so-called “principal engineers” and “engineering leaders” is their adamant refusal to make doing the Right Thing (TM) sexy.
Your employees are monkeys: act like it.
..several years later..
Escalating cloud costs, high staffing cost, staff turnover, heavily reduced margins, decreased productivity, burnout, clients unsatisfied, C-suite paving over this by hiring more marketers...
Salesforce, not exactly a small monolith company, did this for a very very long time.
The company eventually folded[1]. Turns out the company was burning millions of dollars in hiring + infra, while generating only $600,000 in revenue.
[1] https://newsletter.pragmaticengineer.com/p/the-scoop-fast
FTFY.
The magpie was practically furniture (Over a decade there). We speculated that he had buried a literal body for the CEO based on what he got away with. Shiny objects was an astute call on the part of IT guy (he was setting up another new MacBook for him)
Actual, production experience, is, IMO, a requirement to make decisions. No-one will make a good decision to ditch or embrace, say, microservices, based on a HN conversation and a few blog-posts. Nor will they make such a decision based on papers in science journals.
But rather based on failures with monoliths, failures with microservices, successes in Rails, successes in Drupal, and failures in Rails or Drupal. (Or leptos, React, flask, whatnots). Actual felt pain, and drawn learnings. Actual celebrated successes and drawn learnings.
edit: I'll often bring that up in consultancy. "Yes, Rails is great because X and Y. But Z is a rather real danger for your case; we've been bitten by that when building FooBarLy..."
What I'm trying to say: yes, indeed, this person is also collecting experience and evidence to make future decisions on. That there's a real need, and actual benefit on trying and implementing new tech. If only because otherwise we'd still be maintaining COBOL mainframe spagetti (oh. wait...)
I’ve been applying to jobs for months and they’re all looking for go and python devs.
I have production experience with both languages, their common web stacks, and many others (ruby, js, php, c#, elixir, erlang, rust).
I’ve felt that even mentioning that I have experience with other stacks is a turn off to recruiters and EMs.
Nobody seems to care about breadth of experience nowadays.
But I guess we misunderstand each-other. None of them cared that I knew "a lot of stuff that isn't appropriate here".
For example, a recent gig, hired me because I'm not just another Rails "expert", but a Rails expert with Typescript experience, who built large CI/CD pipelines with containers and has built complex (is there another way?) AWS infrastructures etc.
Sometimes they need someone with that exact skill-set. In this case, they needed someone to help them move from yet another "upwork-delivered-rails-spagetti" to something that could actually be maintained.
I convinced them to drop the react/typescript frontend for now (it was terribly bolted on) and to forego building their own PaaS nightmare on AWS but instead just push to Heroku - for now.
My experience helped them make tough decisions.
Sometimes gigs hire me because I have a weird combination of experiences. But more often because my experience allows me to help them make decisions on architecture and such. Granted, I am often hired as "senior architect" or some such. And one of the first things I do, is convince them they should never again hire an "externalm interim architect", lol.
Feels like because I didn’t work in go or python at my most recent job, I’m having trouble landing anything. (I have 5 years of golang prior to my most recent job)
My experience working in devops doesn’t seem to matter for SE positions either.
I give talks on meetups and conferences, so my insights are seen.
I prefer to work in start- or scale ups, so there's a real need for some "know it all".
It might just be the area I live in, but I wouldn’t even know how to start looking for freelance roles that aren’t just poorly paid 3mo junior contracts.
Just not for your prod architecture. Many late night beepers have beeped and phones lit up because the piece that holds together the two pieces that let that thing talk to the database queue monitor thing stopped working so the whole thing went down.
> I'm guessing this guy was a serial job hopper who had no expectation of being able to progress up the ladder at the company you were at.
In many places no one will invite typecast folks to transition to different, more interesting roles that align with their interests. Instead the person will simply be discarded when they're no longer needed for that thing they do. To get around this requires some initiative and that means not "asking for permission" to try new stuff. Sometimes it's better to just take a chance and do something new. There's a risk of cargo-culting, of course, but hey there are worse things that can happen.
Danluu, as he indicated many times, comes from workplaces where staff are paid in multiples of 100K. These are elite "end-game" jobs, not "dead-end" jobs. Such staff are very much tied-in to the performance of the company objectives (in a real sense ($$$$) not in a mission-statement sense), so yeah, these places ALREADY have resources and tech in place that are marketable in other places. There's no need for folks in those workplaces to desperately get out of some php dungeon run by a B.O.F.H petty tyrant.
It's like changing to the new version of an OS on day 1 verses waiting 6 months.
But engineering management is so busy filling out TPS reports they don't have time to actually do any oversight.
So, not fun, not rewarding, no intellectual challenge, no career benefit. Why exactly should I want to do it? This isn't the goddamn United Federation of Planets, nor is the company a church - why exactly should I go above and beyond what I agreed to in exchange for my salary? It's not like the bosses go above and beyond either, nor do they believe in company "mission".
To be clear: I understand the importance of actually doing your job right, and benefits of using boring tech, but you are not selling that well here. Employees need food and shelter and creature comforts, and so do their families. They are going to think beyond the current job, because if they won't, nobody else will.
How does easy work affect scale?
1. It's hard to estimate what will or won't be easy to maintain down the line;
2. New tech becomes hot new tech because it promises to be easy to maintain down the line;
3. Most of the maintenance burden is due to choices of other people anyway, so you have limited control over that.
Trying new tech is often a bet on the promise of better maintainability turning out true. It usually doesn't, but the status quo is so bad already that people are grasping at straws.
A lot of times tech is so focused on the tech that they forget about real problem they’re trying to solve.
Because you're a professional, and part of that means doing things to help your team succeed.
> They are going to think beyond the current job, because if they won't, nobody else will.
This is also right, and a good thing to hold in the other hand.
Reconciling these two forces needs a healthy org that allows employees to grow, along with a recognition that sometimes the forces conflict, and sometimes they don't. All we can do is play the cards we're dealt in the best way.
If you really want to learn new tech, that's what the off hours are for. I say this as someone who has a lot of things that intrude into those hours. I'm (slowly) learning frontend after being a backend/compiler dev for a long time. It's...not easy, but I like it!
The career benefit to me is that a simple tech stack allows a company to move fast and prosper. A prosperous company is usually financially rewarding even if it's not the most mentally rewarding.
Getting tangled up in shiny new toys can harm your ability to move fast and it can have a negative effect on your career at that particular company. Especially since the shiny new toy today is old and rusty tomorrow, but boring stacks will always be old reliable.
If they are not, I am truly sorry.
If it's all above board, and the non-aligned parts are agreed-to, all is well.
It does help you get the next job. You’re just pitching it wrong.
Instead of “Built boring tech” try “Delivered $5,000,000 return 2 months early”. Watch your inbox blow up. Business leaders don’t care about what you do, they care about results. What you do to get those results is just an unfortunate cost and obstacle to overcome on the way to the outcome.
That's the reason why CS graduates with only bells and whistles in their CV have hard times getting a relevant position - glitter over your resume doesn't deliver value at all.
If this is true, why does everyone still think that filling up their Technology Bingo card will get them their next job, rather than delivering business value?
It does for entry and mid level jobs. When you do this you’re advertising that that’s what you’re looking for – a grunt level job.
Unfortunately most job seeking advice out there is for people new to the industry. Because there’s more of them.
Think about it this way: Would you trust a CEO whose resume lists word, excel, powerpoint, and google docs under skills? Probably not, but you sure would expect a ceo knows how to use those.
How do I do that without lying through my teeth? 17 years on the job, I never had the data to even begin estimate that kind of things. It was never my job to know it (I'm a programmer, not an accountant), and it was often actively hidden from me.
And how did you do it? How did you get your numbers, and what did you tell recruiters when you didn't?
Might be more common in companies managed by OKR where you always know the business impact of your work. The business impact is your prime objective and you’re free to figure out the implementation.
Because, IME, managers, etc. love it when you show an interest in how the business works and where your part fits in. It also makes their job easier if they can relate the crappy stuff they have to assign you to how much benefit the business gets from it.
This also happens after explicit requests for feedback. Maybe they didn't actually meant it and I'm supposed to "take the hint" or some neurotypical bullshit, but when I hear such requests I tend to take them literally, and provide real feedback on the real issues. Unfortunately for all of us those tend to be stuff that ossified years ago and cannot (or will not) be fixed any time soon. Ever, in my experience. Quite the downer.
Last time this happened I ended up being axed from a wave of layoffs. Whether their short term workflow and subsequent technical debt finally caught up to them, or their parent company just wanted to cut costs, I will never know. I do know my technical expertise was highly praised, and middle management felt I wasn't quite aligned with the goals of the company, whatever those were. (I think one significant cause was that I only emailed them about significant problems, and kept to myself and my team lead when it was smooth. I think in the future I will stop trusting them with anything negative.)
So yeah, I can bet they love it when you act interested about their work, but start questioning their decisions (even just the one directly related to your own work and its likely impact on theirs), and the tone of the conversation changes very, very quickly. At least where I've worked.
I'm not privy to any discussions you've had, of course, but I will comment on this because I see it in many tech people: don't question people's decisions. No matter how you phrase it, it will appear critical. And people (neurotypical or otherwise!) don't like criticism, no matter how much they claim to welcome it.
Those "explicit" requests for feedback? They want positive feedback. If you really must say something negative, suggest the opposite in a positive sense instead. i.e., instead of saying "we should not have done X. It was a bad idea," try saying "I think if we had done Y, things would have worked well" while avoiding any potentially embarrassing references to X.
Also
> neurotypical bullshit
Be careful that you're not "othering" the person you're talking to and dismissing their concerns because that's what I think when I read this.
When someone asks them feedback so they can improve, I tend to concentrate on what could have actual impact, and this heavily biases towards negative feedback: the build system is bulky and not well documented, newcomers waste time on it, we should replace those global variables by messages if we actually meant to do an actor model, this piece of code is 10 times bigger than it needs to be and will drag us behind if we port it as is…
To be honest, phrasing the above in a nice way that avoids hurting people's feeling is exhausting. They want to know how to make their system better? Well, here are the parts that suck the most, addressing those should have the biggest positive impact. Short and to the point. Oh you didn't want me to tell you that? Why did you fucking asked then?
Sometimes, often, feedback is about timing. The time to talk about how better to architect a thing is when it's being architected, not 2 months later. Instead of "This is crap" try "Hey I know this works for now, but there's room for improvement here. Next time we're talking about a system like this can I help with design? I have a few ideas what we might improve".
Volunteering to do the work of implementing your improvements goes a long way to making people more receptive to feedback. Otherwise you're just giving people homework and they almost certainly already have enough of that.
edit to add: Yes, creating change is work and it is exhausting. That's why organizations value people who can pull it off.
That is to say: You can start now. There’s always plenty of new mistakes to make.
But there's worse: most of the time they already know. The various issues I balk at were often bothering them for years, and yet fixing it was never the priority. Week after month, there always was something more pressing, generally about some deadline. Such issues often have gone long enough that the cost of keeping the current cruft have long exceeded the cost of replacing it. But this cost is gradual, so the fix keeps getting pushed further and further into the end of times.
Perhaps my biggest mistake is not seeing that such organisations are beyond helping. I keep getting fooled by promises of doing things right going forward, when I should instead extrapolate the past. Either they're already work in ways that I can approve, or at least live with, or I should seek something else on the spot.
Yes, building a solid project with boring technology that delivers real business value sounds good in theory but not so good when applying for a new job. Maybe it can help after you somehow manage to pass some initial screening.
At a certain point in your career, you'll realize that the business manager can override any technical hiring manager. Because at the end of the day delivering results is sexier, than bells and whistles in your resume.
Saving a company from political mayhem is a pretty good achievement to have on your resume. It's also impressive because most engineering teams give up early on.
The problem is that $100mil is all pixie fairy dust when you're working on a new project. I wish this wasn't true but it works out better for you to implement it as costly and complex as possible, show off how smart you are, then simplify it during a cost cutting initiative (wow they must be so smart to make such an obviously complex system so simple).
The secret is that while you think you're getting away with something playing this game you're actually doing exactly what the business wants.
How so? I would think the business wants to spend as little money as possible.
Your floor is limited because you can only drop your costs to zero, but there's no ceiling on how much revenue you can make.
We don't hear of such failures any more because software projects (or products) no longer "fail" in the traditional sense -- they turn into endless money sinks of re-architectures, re-platforming, tech debt repayment or expensive maintenance, that can continue as long as the company has cash. When the company does run out of cash, it is difficult to say to what extent tech expenses or lack of revenue due to slow software delivery played a part.
Good luck having the opportunity to work in a project where you have even the faintest idea how much money your contribution will make or save. I don't know about you, but never in my 17 year career have I had enough information to even attempt computing these numbers. And even if I could have, it was never part of my job description.
So how did you know your numbers? Or if you didn't, how did you made them up for your interviews?
"Your manager tells you what to work on" isn't how big tech runs for the most part, in fact it's a bit sad to think that some people work like that
The empowerment you speak of is but a faint dream in my town.
Not to mention the monetary impacts of decisions totally outside my control. I can tell the architect "you suggest A, but B is simpler to use and makes the API 3 times simpler at no loss of functionality", what's the point of estimating the impact of such a no-brainer when the architect answers is "you're correct, but we'll do it my way" (real story)? And how do you expect me to estimate the monetary impact of pointing out that their TPM provisioning is missing a verification step? That stuff happens inside the factory, a problem at this stage is unlikely anyways. And even if I could somehow divine my monetary impact, the best I can say now is "I did good work for this company, they didn't listen to me, and now they're going under". Not even kidding, they are going under. I just ended my gig there because I couldn't take it any more.
What are those wonderful places you worked at where you could estimate your impact with reasonable accuracy?
For example, build system improvement on 10% of build time across 200 developers who on average get paid 300k a year - that's a very easy math, no ?
Same for time to deploy, improvements on time to fix a bug, etc. etc.
You can extrapolate and compare initiatives and projects on t-shirt sizes and ROIs. Knowing where yours sit as well.
What places I've worked at ? Mostly business that made some money and were stable.. apart from a start up that was VC funded and made no money
Ground software for an observation satellite. My internship was about implementing a dead simple neural "network" (2 hidden layers, no feedback), everything was specified from up top, we didn't even get to touch the learning algorithms. Impact? I guess a big flat zero, since all the differentiators was in the learning parameters.
Peer-to-peer social network before Facebook. Never made a cent.
Geographic Information System for the military. I was for obvious reasons not allowed to know enough to estimate the impact of my work. And even then all decisions was made by the customer, and once the user (a different entity) saw the Rube Goldberg contraption we dully made for them they predictably balked, and we did what we could from there. Which was, not that much. I did some useful stuff for sure, but mostly I participated in a system that was arguably worse than the one that preceded it.
A visualiser for civil radar data. Data in, little planes in the screen out. And other nice stuff. I designed a simple C++ API that allowed the client to write business code faster than we would have ourselves (if only because of communication overhead), saving weeks of work. That contribution was utterly ignored for personal reasons, and I was eventually out. I have no idea what my actual impact was, because I don't know how far the project even went, and how widely it was eventually deployed.
The maintenance of ground software for small civil observation drones. I did some cool stuff, but then was asked to transfer ownership of this software to a recently bought team (that did stuff similar to the company I worked for). I could have known how many drones were actually deployed, but to be honest my thing just saved a few minutes of flight, while most of the cost is to get the drone and its operator on site. That company was never really profitable, I hope the good people I met there are doing well.
Scripting language for a programmable logic controller test environment. For the military, so I don't think I was allowed to even know the size of the team we'd deliver the software to. I got good feedback from them (they were happy about what I did), and I'm pretty sure my static typing made things easier for them than if I had just picked Lua or something, but how easier, and how much money it will save in the long run I have no freaking clue.
Stuff in a missile company I cannot disclose. I believe my impact was almost nil, I couldn't stand their abysmal tech environment.
Maintenance of a ground troops training system (a glorified laser tag, with debrief helpers and so on). Good luck assessing the impact of a better system, and I was just asked to do small tasks I could barely chose anyway.
Prototype ADAS system. It was never deployed. Actual impact was therefore basically nil. Cool stuff to work on though, the CAN bus is a think of beauty. One of the rare instances where I could actually learn from example, instead of seeing yet again one of the gazillion obvious ways how not to do stuff.
Ground software for some IoT device. Impact fundamentally uncertain, we had yet to sell it to anyone.
Incident reporting software, based upon a more generic distributed base. I made the encryption layer (between users & company server), with a security based on PAKE (thus avoiding a PKI, which simplified the work of the sysadmin, at a slight loss of security). Impact fundamentally uncertain, we had yet to sell it to anyone.
Charging stations for electric vehicles. I did the TPM provisioning, and mentioned a low-key security issue along the way. I participated in a questionable micro-service that was meant to help user interfaces (yeah, their IoT stuff had a micro-service architecture). Impact: whatever I did didn't save them: one year after I left, they're now going under.
Preliminary study on the possible use of AMD-SEV to prevent users from peeking at our secret sauce (DRM). I don't think I was allowed to know the list of clients, and it's not even the only alternative. I don't think I could ever have assessed the long term impact of my work there.
Flight recorder for trains (not a flight recorder then, but you get the idea). I just did little tasks here and there, didn't get the chance to have a good bird's eye view of the thing or its environment. Deployment base was knowable, but the business impact of my work was likely minimal, beyond "finish this step so we can show the client we're on track for the next short term milestone". The whole thing is a heap of technical debt, common components are impossible to update (user projects aren't locked to a given revision, they all pull from trunk), the build system is a home made monstrosity that doesn't help more than the standard monstrosities (I hate build systems)… and I was just axed from a round of layoffs.
Cryptographic library I did on my free time: https://monocypher.org/ Nice little thing with a significant user base in the embedded ecosystem (not even my primary target). I controlled everything from start to finish, and I have no idea how many users I have, let alone how much time and money I saved them. In part because it is so simple, with such an outstanding documentation (which I mostly didn't write), that most users don't even have to bug me.
---
To sum this up, my resume looks fairly horrible with respect to what I know of my actual business impact. Most of it, I think, was entirely outside my control. And I don't think I'm exceptional in this.
Make it a point to gather numbers like e.g. revenue figures, growth rates, costs, usage metrics. It is true that us underlings aren't usually handed these numbers, but you might be surprised by how easy it is to get them once you start looking. And once you have those numbers, you can derive some good estimates around your own impact towards the business.
Having those numbers will help you understand your own impact. They will also help you avoid companies that, for a lack of a better word, are destined to fail.
(I'm a fan of your blog btw. I really liked the article on the Monty Hall problems!)
I should be able to try this the next few weeks (new gig). But again, I'll be working on a prototype where such projections have already been made, the plan have already been laid out, and I'll mostly have to follow orders with relatively little wiggle room (they also expect me to hit the ground running). My only expected impact there is whether the stuff gets done on time or not. And assuming it is, the main company will throw the prototype away and rewrite everything anyway (at least they're supposed to).
If it's not intellectually challenging, you're not working on interesting systems. If you have to find interesting tools to find intellectual stimulation, consider a different system to work on.
As an example, I got to learn astrodynamics as part of my last job. Maybe not intellectually stimulating to everyone, but it didn't require me to learn the latest tooling just an interesting area of math and physics. The tooling and the language for the software wasn't that interesting, but that's fine.
- Environmental chemistry
- Mass balance simulations
- Wastewater treatment processes
- Geotechnical engineering
- Ecology
- Mass transit systems
And quite a bit more. I could not agree with you more. Even without the broad range of interesting subject matter, there's no end to intellectually stimulating work simply trying to architect a system well.
If you're asking on a personal level, I think that if you keep to the high ground, you're more likely to find long-lasting happiness. David Graeber spends a good deal of the pages in Bullshit Jobs on this topic.
Your goal isn’t to be intellectually stimulated at your job. If you want that, read a book. Your job is to deliver reliable, lasting value.
Overcomplicating the architecture for the sake of job security is a con you run on your employer.
And then people are surprised burnout rates are as high as they are. Lack of mental stimulation leading to burnout is the white-collar equivalent of repetitive stress injury at jobs that put strain on the body.
> Your job is to deliver reliable, lasting value.
Nobody is actually paying you for that. In fact, it's probably counterproductive to the business goals.
> Overcomplicating the architecture for the sake of job security is a con you run on your employer.
On the other hand, "work ethics" and professionalism in modern workforce is a con your employer runs on you. The further above and beyond you go, the more work they get out of you for the same pay.
Yes, I'm being a bit obtuse here. But my point is, there needs to be a balance. Or at least a mutual understanding of conflicting incentives. We can't demand facets of professionalism in the way that benefits employers short-term, but deny and scorn those that empower the professional. Independent learning and broadening one's experience is a part of what being a professional means.
That's not my goal. That's not even what my employer wants most of the time. Most of the time, it's just about a bunch of rich dudes (and fewer ladies) wanting me to make them even richer. That's how the system works, no need to write the "C" word, or call me with the other "C" word just because I say it so bluntly.
My goal is to enjoy my life. I have various ways of enjoying life, many selfish, some altruistic, very few aligned with the will of the rich people up top. My job takes about a fourth of my waking hours (if I worked full time it would be a third), valuable time that I'd rather spend for me and my loved ones, instead of giving it to people who already have way too much. The only reason I can sometimes tolerate unrewarding work is because I don't have a better way to pay the bills.
The reason I don't over-complicate architecture isn't because it will make more money for my employer (sometimes it means making them less money, especially in the short term). I keep it simple because I can't stand wasted effort.
I tend to think that cargo cult programming and resume-driven development are the intellectual path of least resistance. Perhaps it's analogous to, "I'd rather rewrite this than understand how it works", because that requires less intellectual effort. Quality engineering is not achieved by the intellectually lazy, from what I've seen.
Proof: that's literally what a significant positive growth rate of an occupation means - if the doubling period is N years, then at any given moment, half the workforce has N years of experience or less. I don't remember the estimate for webdev, but I think N was something between 3 to 5 years.
The trick is to get the Project Management to migrate to a hot new framework: Vanilla JS...
Sounds like "how should I know what I think before I hear what I say" ;)
Point is, there is more to the equation. Employees don't exist in isolation, and when jobs actively refuse to take into account that the workers are real, living human beings, with needs and plans and dependents to feed, then resume-driven work will continue.
wait, let me guess, if I truly loved my wife I'd also never play a video game by myself.
Watch as my eyes roll out of my head, onto the floor, and out the door.
I think this is an absolutely right read on the situation. To put it in a slightly different context, the magpie developer is more akin to a "sociopath" from V. Rao's "Gervais Principle" [0], doing the least amount of work for the company while forging a path forward for their career. In this case, it just happens to not be within the same company.
[0] https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...
I did not sell it well, that's fair.
>> Why exactly should I want to do it? This isn't the goddamn United Federation of Planets, nor is the company a church - why exactly should I go above and beyond
HN doesn't want to hear this answer: you do it for the PEOPLE around you.
If you build sexy tech, and then get sexy job and I have to clean up your turds... well you can go fuck yourself. Hope that I'm never going to be the one answering the linked in request for a recommendation or sitting on the other side of the table when you come in for an interview.
The tech job market is bad, and getting worse. You're gonna need an advocate on the inside if you want or need work quickly. That means former co-workers and bosses. NO one is gonna hire clock puncher who did selfish resume building project and left. Dont be that guy.
I think that delivering a solution which works, even if it is not sexy, is exactly what one agreed to in exchange for one’s salary. It may not be fun, it may have no intellectual challenge, and it may have no career benefit, but it is rewarding: the reward is the salary.
if you have people who are effective, allow them some space for fun and intellectual challenge even if it takes a bit away from the workload - if you disregard those human factors something will give up at the end, perhaps catastrophically as efforts are made to add "sexiness" to the core of the mission critical workload
Don't forget that simple isn't easy. I find it very rewarding and intellectually stimulating to solve a problem with a solution which is as simple as possible - but no simpler.
My very lonesome role in these cases is to insist that this shiny thing is no standard at all and that our users would be quite happy without it.
The Wall Street Journal Guide to Information Graphics is an essential reading for anyone who regularly presents data.
I've always felt that the magpie syndrome you describe is because of the desire to build new things, rather than maintain what's there.
I watched a repair show recently where a craftsman repaired an old 70s bedside clock. The pride, time and patience he took in repairing the clock was wonderful to see, actively avoiding rebuilding parts if he could reuse what was there, even if there was a crack or blemish.
I've always respected engineers that maintained and fixed software well, and also knew when to reach for the right tool in the toolbox. Better yet, those that knew when not to build something new. Perhaps that's something you learn through experience and doing, but I wonder if it's actively taught, encouraged, or rewarded in the workplace. It really should help get you your next job.
Remember that your resume will not hold much value, when you give off "we built this thing with friends in a garage" in your resume and little else.
Have you supported anything in production? No? Explain why should you be a candidate for anything other than an entry level position as a SwE.
That's why you lie about it.
Some stuff got done, but nothing too mission critical that kept me up at night and it was pretty relaxed.
Cases like this always fascinate me. I've led a "move from Data Center to AWS" effort twice, and both times it was at > 50% cost savings. However, I think both were probably small infra footprints compared to many cases like many others.
In software engineering We have all these principles and patterns and whatnot precisely because we have to deal with a pile of things that don’t work well together.
This reminds me of the time that I complained that a sensor on the machine we were developing was way too complicated and that there were far simpler ways to accomplish the same thing.
Someone with more history on the project explained. The engineer who designed that sensor expected, and received, a patent for his work. The company was quite generous with patent royalties and no one's getting a patent for an off the shelf float switch!
Taking the CV aspect literally — this is a sleight of hand because it's a metaphor — I know lots of people who do this stuff that don't have a CV at all.
There's levels to it of course, but I don't really view it be any different to people who insist on using old machines as a bit (but in the other way obviously)
Tech trend will continue until this business mindset of burning money on shiny things changes.
When you see opportunities to do such work in a way that delivers something new and valuable, I recommend taking hold of them. I learned this somewhere along the line, but not until I’d missed a few from self-doubt and indulging the inner magpie.
Clear, simple and value focused engineering is _exactly_ what I’m looking for in candidates these days. Focus on the long term, not the short term — just like with investing.
The most common argument against simpler architecture is, "but that won't scale and we are going to grow to be a valuable company." So the idea that they aren't a $10 million company does seem somewhat relevant.
Complexity for the sake of complexity is foolish.
This exceeds my ability to suspend disbelief.
You seem to be implying that all complexity is the result of an informed decision made by someone but this does not match my experience in the field.
My life would be much easier if this were true. One day, maybe....
"Simple architectures" and "boring technology" are slogans meant to keep technologies like GraphQL out of your stack. Writing less code and getting features for free is how one describes "exciting technology".
Show me your problem, I'll show you how to solve it with the least amount of code and dependencies.
They just read blog posts of what to do.
As an analogy, I’ve manufactured a lot of stuff out of wood, concrete, plastic (3D printed) and increasingly more metal. When I need a jig, I know what each material will be like and what I will be getting out of it.
Yes, it's worth learning a little about clay. Maybe I'll even enjoy it.
(And I know you’re trying to make an example, but I have already looked into clay a bunch of times. Clay just doesn't have the material properties that my projects tend to need. It'd be like learning COBOL expecting to build front-end apps.)
A sobering account was our second CTO who told us that their previous 65000 user application ran on a single database/server, so for the rest of that project we just kept most our application on a single DB and focused on tuning that where needed.
I believe even Stackoverflow used to run on something similar, not sure.
When hiring someone, I love to see if they are visibly annoyed when something is over-engineered. Or when a user interface is too complex.
If they have strong emotional reaction, it's a good cultural fit :)
The inverse is true too. If someone "gets off" to complexity, I view it a red flag.
>since we’re only handling billions of requests a month (for now), the cost of this is low even when using a slow language, like Python [and simple synchronous code], and paying retail public cloud prices.
This is more and more my philosophy. I've been working on a data science project with headline scraping (I want to do topic modeling on headlines during the course of the election) and kept preferring roll your own solutions to off the shelf ones.
For instance, instead of using flask (as I did in a previous iteration of this project a few years ago) I went with Jinja2 and rolled my own static site generator. For scraping I used scrapy on my last project, on this one I wrote my own queue and scraper class. It works fantastically.
We are often embarrassed by the simple.
The simple makes us feel small, unsophisticated, and unimportant.
I always find it amusing when I find a way to solve a problem simply, and my coworkers answer somewhat directly that it seems very simple.
To me that's mark of success, but to many it's uncomfortable.
Of course if we didn't work on it then it's over complicated it needs to be rewritten. Go figure.
> • Our various apps (user app, support app, Wave agent app, etc.) can mostly share one API, reducing complexity
This could of course be done with a RESTful API.
> • Composable query language allows clients to fetch exactly the data they need in a single packet roundtrip without needing to build a large number of special-purpose endpoints
This, too, can be done with a RESTful API. Two approaches which immediately spring to mind are passing desired items as query parameters or passing a template item as a query parameter or request body.
> • Eliminates bikeshedding over what counts as a RESTful API
This may actually be a benefit of GraphQL! There’re a ton of soi-disant ‘RESTful’ APIs which simply don’t follow REST principles at all (e.g.: ‘REST’ ≠ ‘JSON RPC’; also HATEOAS is hugely important, even when the hypermedia is in a non-HTML format such as JSON). But I also think that real REST really is a powerful framework for distributed computing. Sometimes the discussion really is bikeshedding, but sometimes it is a definition of terms.
I've implemented this twice, you quickly end up at something resembling GraphQL after a couple of rounds of "iOS needs the the profile picture in two different sizes, dependent on native screen size, and with version 2 of the blurhash"
Might as well use GraphQL and get schemas, libraries for clients and servers and IDE support out of the box at that point. It's no panacea - many of the backend libraries were somewhat low quality last time I checked - but I wouldn't hesitate to use it if I had a somewhat complex main product served by multiple frontends following different UI guidelines.
I am certainly past the idea that monoliths impose any kind of barrier to complexity. In many ways I find it to be precisely the converse.
Sure, there's nothing meaningful to choose between one giant plate of spaghetti versus a dozen small plates flying in loose formation. I'd still rather have lasagna than either.
SO much yes. You may have the prettiest architecture, and the best separation of concerns, and the most modular modularity, but if you don't have the right competencies, processes and people all that would go out the window before you can even finish the power point slides showing off your nice new architecture.
It's like a garden - if you don't tend to it, it turns into a mess, and the fight against entropy never stops. If you also have to fight the team along the way you've already lost
So much of the content around the programming web is about huge scale and domain-specific stuff. We’re in a pretty magical place where the core technologies we use can scale to the size of a successful business without much work. We should promote that more!
Seems in line with the we're soon going to see a single person billion dollar company narrative.
What's our solution for simplifying our operational platform? It's to pay AWS to do it for us. And now we use Fargate and Lambda for hosting.
Is that simpler? Or exotic and more complicated? There are tradeoffs, which will vary from team to team and organisation to organisation. To one team, slapping functions in Lambda and knowing how to support them are more straightforward than a team of experts who know how to operate monoliths at scale.
For me, the real conversation is about something other than simplicity. Simplicity is subjective. It's about familiarity and competency. Pick what your organisation is familiar with or competent with (or has the ability find people who are...)
Something that seems apparent with handling money in accounting systems (and trading systems) is you can't really do concurrency any way. If someone changes their account you need to lock those database records from access to any other thread and make everything atomic. Otherwise it can lead to race conditions where the same funds can be spent multiple times (this is something that has effected Bitcoin exchanges before quite ironically.) So maybe they use a sequential design because it makes more sense from an accounting perspective and eliminates the potential for race conditions? But then I'm still very curious about how performant such an approach is?
I had a similar experience using async Rust to make a boring HTTP server. Debuggers can't trace across `await`, so debugging was a very slow manual process. Also, I wasted a lot of time dealing with borrow-checker errors.
I finally gave up and tried using Rust HTTP servers that let you write threaded request handlers, but there was only one (Rouille) and it had show-stopping problems. So I wrote a good one:
https://crates.io/crates/servlin
You can use Servlin to make a boring HTTP server in Rust, with threaded request handlers (no async). I use Servlin to serve https://www.applin.dev , running on Render. I'm also using Servlin (and Applin) to build a mobile app.
The article itself mostly discusses tech choices rather than architecture. Half of their choices, they seem to regret.
On the service itself: It seems detrimental for African countries (or any country) to allow external financial services to get ingrained and allow them to milk their own populace and prevent the development of state-owned, cost-neutral, non-profit, long-term solutions for the public good. Payment should be public infrastructure, not private profiteering.
Is anyone here donating Dan $180k/year (+VAT) for his blogging?
It's like the StackOverflow example. Sure, the architecture is conceptually simple, but you still bought servers and are maintaining them yourself. That's pretty complicated unto itself. Probably the right decision, but not the "simplest".
Later, I read a Paul Graham essay where he says that a company should choose its programming language based on how fast they can iterate in that language. And it clicked for me.
This talk explores the tradeoffs: https://www.youtube.com/watch?v=SxdOUGdseq4
It's just a shame that the syntax will forever be a barrier to widespread adoption.
It's an opinionated language: dynamic, with immutable data structures, hosted on the JVM. I like it quite a bit, personally, and it's been adopted in places where it makes sense.
But many-to-most projects are better off picking another runtime, and the syntax has little to do with why.
Clojure ticks neither of those boxes, unfortunately.
I suspect the era of VC-money and ZIRP led to a large number of engineers who were disconnected from successful long-term business outcomes, so there was no incentive to simplify.
In my experience, this exists in a few, rare (investment bank) trading floor technology teams. From a distance, it looks like traders who are programmers.
It's entirely possible to measure teams on business outcomes without having them own things end-to-end, but to be really effective in this they'll need to collaborate with the other teams across the company - which is generally a desirable outcome.
VC-money and ZIRP is a convenient shorthand for what's changed, neither are terms I think particularly encapsulate what did change, but I think it's very hard to argue that it's still just as easy to get a bunch of funding without a business model. (aside from in AI perhaps?)
That said, "simple" and "complex" are subjective terms, and they're not easily defined. Even when everyone aligns on what they are, complexity can still creep in. So it takes constant vigilance to keep it at bay.
In the infinite wisdom of grug[1]:
> given choice between complexity or one on one against t-rex, grug take t-rex: at least grug see t-rex
> grug no able see complexity demon, but grug sense presence in code base
I mention this because I've been a lot of times in a place where instead of adopting "well maintened library that everyone is using since more than a decade" I had to write and maintain my own solution just to appease someone else because they thought it would be less work (it wasn't, maybe for them, definitely not for me).
We need less posts in software development that people try to argue in absolutes.
I agree that this is a deep topic that requires a nuanced discussion, but my point is that I like the pragmatic approach mentioned in the article. You see threads here arguing that GraphQL and k8s are not simple, which is true, but I think it's wrong to hardheadedly avoid these technologies because of their "complexity", than to reach for them when the right time comes to use them.
Linux, Bash, Python FTW. Everything was designed to be as simple as humanly possible. I mostly built it in my spare time too.
I once showed it to a real developer and he was horrified that some long messages that had to be sent were hardcoded into a file and concatenated to the one bit that changes...it was my prototype that worked so well I never changed it.
It only ran every 15 minutes, since all the systems that depended on the order ran less frequently than that. Everything was handled sequentially, and quite deliberately so.
It actually had a lot of functionality it gained over a couple of months, it had to print some stuff, detect fraud, fix addresses etc.
My favourite bit is that several companies had offered to build it for £mega and decided it couldn't be done! I wish I could describe it in full...
But on a more contextual note: I would contend that microservice architectures are in effect much simpler than monoliths. The UNIX philosophy of small tools that do one thing well is still very applicable. It's easier to deploy small changes quickly than the massive deploys that could take significant amounts of time. Deploying to a FaaS is quick and easy for almost all things.
If your product requires complex coordination/orchestration just as an operational modality then I'd question the value of that product design regardless of architecture type implementation, even for web-scale products. AND this is really a different discussion.
The UNIX philosophy was conceived in the context of a single computer—with dynamic linking the cost of splitting out a new binary is close to zero, so making each module boundary an executable boundary makes some degree of sense.
In the context of a cloud computing network, there's a lot more overhead to splitting out a million tiny modules. Cloud service provider costs, network latency, defending against the possibility of downtime and timeouts—these are all things that UNIX programs never had to deal with but that are everyday occurrences on the cloud.
The trade-offs are very different and so the UNIX philosophy is not equally applicable. You have to weigh the benefits against the risks.
Don't get me wrong - this can be taken to an extreme and from my experience the places where MS seems to fail (the service is not used, or hated by the engineering teams) is in scenarios where it was absolute overkill.
> The trade-offs are very different and so the UNIX philosophy is not equally applicable.
There is a reasonableness standard that exists in both contexts and applies to both in the same way.
It applies to both in the same way in that there exists a definition of reasonable and exceeding it would be bad. But those definitions of reasonable are going to be completely different because the environments are completely different.
I guess my point is that invoking the UNIX philosophy as such brings to mind mostly tools whose scope would frankly be far too small to justify deploying a whole cloud service. It would be better to treat deployable units on the cloud as something new, rather than trying to treat them as if they were a logical extension of single-node UNIX executables.
However, I do wonder if things like Modal allow you to focus only on writing business logic. No dealing with deploys, network comms, memory, etc.
(No, I don't work at Modal, just wondering "what comes after monolith" -- Modal, Temporal, etc. all seem quite good)
I'd love to know how you ended up combating this! I'm assuming something like graphql-crunch[0] or graphql-deduplicator[1], but I'd love to know what worked well in practice.
The code is structured in such a way that you only start the sub services you need. We have node that launch almost all services, some only a few.
If we need to scale just a particular part of the system we can easily just scale the same node but configuring it just for the sub services we need.
What's described is far from a simple architecture. Synchronous python? Kubernetes? I'm guessing there's a legacy monolith with a few microservices or load-balancers? Then... GraphQL... why? This is just a hodgepodge of stuff in dire need of actual refactor.
Everyone wants to argue that yes, they ARE going to need it. Maybe not now but someday, therefore they should have it.
Always reminds me of this scene from Arrested Development.
This is why architecture is hard. It is about making the decisions in advance that you will regret getting wrong.
How big of a userbase and what made it particularly difficult to refactor?
However we have > 10 million lines of C++, and the change would have been a major refactoring of most of them.
But.
The described use cases have homogenous workloads. Hence simple architecture. Extrapolate the rest.
And of course there are for sure cases of people increasing complexity for the sake of being needed. That's a topic for sociologists/psychologists to reason about, not for engineers.
- the CLI
- the cost model
- the cost optimizations
I'm tired of this endless looping. A cheap VPS + Docker is enough and in most of cases (it's even easier to scale/migrate)
A repo for not hyped, dead simple solutions, to work with.
In defense of simple architectures - https://news.ycombinator.com/item?id=30936189 - April 2022 (189 comments)
I've solved a lot of the issues he brings up - gevent, no orm - well-known transaction boundaries, etc. It's pretty pleasant to work in.
Meanwhile adult sites probably run on some boring php and serve orders of magnitude more content (harder to cache) while being free and top visited sites in the world
Loved this line:
"I’ll discuss a few boring things we do that help us keep things boring."
Maybe that is why everyone goes with the the latest and greatest sophisticated techniques that are popular on the conference circuit. At least it has a diagram.
For example, virtual machines are something I and probably most other people consider "simple". They're pretty much just little computers. For the most part, I can just work with VMs the same way I would work with computers. No sweat.
My DevOps pedigree, however, also means I consider Docker containers and even some light k8s stuff as pretty "simple" - when these two things feel to other people like massive overcomplications.
On the other hand, I'm pretty new to GraphQL, and it feels to me vastly more confusing than good old fashioned REST API endpoints, where I can crack open the code and actually see exactly how that particular endpoint works. I am mostly alone in my org when it comes to this feeling, because I haven't digested the abstractions that make GraphQL work as well.
I don't really have a good answer then for what simple is beyond "I know it (kinda) when I see it".
The easiness is relative (as you described) and depends on the things you are familiar with. For example, Docker containers and k8s stuff is easy (for you), and GraphQL is hard (for you).
The simplicity should be assessed (somehow) more objectively.
"A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system."
It's quite funny how unknown this quote is. Imagine all the developers that would be out of work if we had simpler systems. It's hard to admit, but developers get more jobs when their work is complicated, it's the hostage taker syndrome.
Software as a thing, is pretty recent, it's important to remember it has a large Silicon Valley influence, and software quality is just unregulated because of freedom of speech warriors.
The software we have today, is just the combination of good government funding and marketing maniacs.
Fintech targeting Africa
> Despite the unreasonable effectiveness of simple architectures, most press goes to complex architectures
We're over-simplifying these terms. What's simple in one way is complex in another way. It makes for great blog posts to use generic terms like "simple" and "complex" for broad-ranging and highly technical topics, but it means nothing. Be specific or you're just tilting at windmills.
> The cost of our engineering team completely dominates the cost of the systems we operate.
Could this be because you decided to invent a new kind of truck, thus requiring lots of costly engineers, when getting two mechanics to weld a crane onto a flatbed would have done the trick?
NIH is rampant in tech. Everybody thinks they need to invent something new when they can't find off-the-shelf parts that do what they want. When in reality, just making their system a bit more complicated in order to accommodate off-the-shelf would have been faster and cheaper. Sometimes more complex is simpler.
Later on in the article Dan mentions build vs buy, but it's not that simple. You can buy and then modify, and you can build in a way that's cheaper and easier, if uglier and more complex. Design and engineering decisions aren't binary. There are more options than just "only use a complete product" vs "build your own everything", and more business and engineering considerations than just time to market and "scale".
> Rather than take on the complexity of making our monolith async we farm out long-running tasks (that we don’t want responses to block on) to a queue.
See, here's the thing with "simple" vs "complex". It's not "simpler" to use a queue than async. Both are complex. Queues may look simple, but they're not. Async may look complex, and it is. You didn't pick simple over complex, you picked something that looked easy over something that looked hard. You made a bet. Which is fine! But let's not pretend this is somehow good design. You just ignored all the complexity you're going to run into later when your queue becomes a problem. Maybe it won't be a problem? But maybe async wouldn't have been a problem at scale either. "Maybe" doesn't equal "good design", so let's not pretend it does (re: the idea that it's simpler and thus better).
> Some choices that we’re unsure about [..] were using RabbitMQ [..], using Celery [..], using SQLAlchemy [..], and using Python [..]. [...] if we were starting a similar codebase from scratch today we’d think hard about whether they were the right choice
This is a sign of a lack of engineering experience. I don't mean that as a diss; I only mean that the correct choice (of these options) is clear to somebody really familiar with all these options. If you don't have that experience, you need to go find someone that does and ask them. This is one of the best reasons why you don't want to engineer something yourself: you probably aren't the best engineer in the world, and so probably don't know the kind of problems you are going to run into. However, if you are engineering something yourself, and don't know the right answer, you need to go ask somebody (or many somebodies). If you were inventing some new kind of device that nobody had ever invented, maybe nobody knows? But you probably aren't doing that, so somebody does know. And if for some reason you can't or won't go ask somebody, then you should use the thing that you[r team] is most familiar with and already know all the pitfalls of, regardless of nearly any other downside from its use. Unexpected problems are a much bigger thing than expected problems.
> By keeping our application architecture as simple as possible, we can spend our complexity (and headcount) budget in places where there’s complexity that it benefits our business to take on.
And this is still good advice. But, again there is no such thing as a binary choice of "simple vs complex". It's all shades of gray.
The rest of the article goes into why the above is not the case.
The language was chosen because it was the CTO's pet, not for simplicity's sake. It wasn't the right choice: "its concurrency support, performance, and extensive dynamism make us question whether it’s the right choice for a large-scale backend codebase"
Synchronous/blocking wasn't chosen for simplicity's sake - the async libraries were buggy! To work around the performance issue:
1) a "custom protocol that runs on top of UDP" was written. No thanks.
2) putting work onto a queue. Event-sourcing, anyone?
> we’re having to split our backend and deploy on-prem to comply with local data residency laws and regulations
It's good that the software was a monolith, otherwise it would have been difficult to split apart /s.
Software is incidental complexity and essential complexity. If you shun certain things as 'too complicated', but they end up being in the essential complexity bucket, you're just going to be building them yourself, slower and with more bugs.
Imagine how different the article would have been had they picked technology whose concurrency, performance and typing worked in their favour.
Combines excellent concurrency primitives, very good performance, and a deep reservoir of packages and libraries.
Of course it's almost never worth switching languages for an existing product, you just get left with a mess of half-old and half-new tooling and invariably get left having to support both languages.
In C# Microsoft often provides the most popular tooling for common problems. For instance, in web development, Microsoft publishes ASP.net and Entity Framework. In Java there is Jax-rs (JSR 339) and JPA (JSR 317).
So depending on your preference for a strong centralized environment provided by a single vendor, or a decentralized standards based solution with many vendors, you might lean towards Java or C#.
Python wasn't a bad choice but even several years ago it wouldn't even remotely have been the only viable choice, and if you ignore "what the people involved already knew" probably wasn't the best. In real life, you can't ignore that and it looms large in any real decision, so I'm not proposing that should have been the metric. But Python's disadvantages can be pretty significant. "Dynamic typing" and "deals with money" is a combination that would make me pretty nervous. YMMV.
This is an interesting comment. I do not disagree. In your opinion, if not Python, then what languages would be acceptable to you?
And that's sticking to reasonably mainstream languages. F#, D, Scala, Haskell, Lisp with strong typing around the money parts (or Clojure), probably a couple of others as well. These have problems with having smaller communities and being harder to hire for, but I'd bet that pretty much all of them still have well-supported libraries for nearly everything a financial company would need to do. (And that you could expect general open source libraries for. Super-specific internal financial protocols may not have libraries in general, or may only be in Java, etc.)
Also I make no exclusivity claims that these are the only appropriate languages, so if you've got a favorite statically-typed language feel free to just mentally edit it in without asking me how I could possibly miss it. (I'm not familiar enough with the Objective-C family to have informed opinions about it, for instance.)
Dynamically-typed languages with progressive typing used rigidly around the money might be OK, but progressive typing makes me nervous and really seems like a bodge in a lot of places. Reading people's experiences with them, even when the person writing the experience is nominally positive on the experience, has not generally impressed me.
I'm writing a lot of email-related code right now, porting something out of Perl into Go, and because this code supports Google-style "+" addressing, where "anemail+something@gmail.com" and "anemail+other@gmail.com" need to be tracked as separate recipients (as defined by the SMTP protocol) but I also have a slate of functions that need to operate on the mailbox, as identified by "anemail@gmail.com", using some types to keep the two things straight has greatly cleaned up the translated Go code, whereas in Perl it was all just strings and there are constant repeated snippets of code trying to (inconsistently!) translate between the two on demand and with no compiler support as to whether it is being done correctly. I wouldn't be surprised this fixes longstanding and hard-to-diagnose bugs simply by the act of imposing a static type system on top of the mass of dynamically-typed goo. I'm fairly sure I've identified at least one place in the old code that it failed to do the conversion, but in dynamically-typed languages it's hard to even be sure about that.
Statically typed languages do not automatically or magically make dealing with currency easier and safer than dynamically type languages. A static language's compiler will not peep as you pass around an "integer" that represents US pennies, or maybe Euro pennies, or maybe Yen, or maybe Italian Lira as of March 23, 1981. But if you have a need to do so, and a need to do it safely, static languages give you much more rigid and safe tools to do it with.
(You can create a "currency" type in dynamically-typed languages that is a lot smarter than an int, certainly, but you can't use the interpreter to rigidly enforce that it is used correctly, and the various incremental or progressive typing options I've seen generally fail to impress me. The cultural temptation in a dynamically-typed language to allow functions to also just take an int and then "guess" the rest will be overwhelming based on my experience and it takes a strong developer lead to push back against people sneaking that in.)
While you're right that only small bits of the code may be doing calculations, a lot of the app may be dealing with money in some sense. In a statically-typed language used correctly, the money values can at least pass through those parts of the system unambiguously, carrying exactly what they are, and the parts of the system doing the calculation doesn't need to do a lot of defensive programming to ensure that the currency values are actually of the expected type. Code pulling these values out of the database should put them into the correct type, code displaying them to users should be using this information correctly. Not just things "calculating" with the money but anything that touches money values, passes them along, serializes or deserializes them, all of it is cleaned up by having a precise specification of what it is that is maintained all the way through the app.
Microservices are simple. Request handlers or queue listeners, transformation and data layers, caching, database connection pools, clients to other systems. They're easy.
Monoliths are where you come across some of the most godawful complicated architectures. Unified object-relational mapping layers. Shared cache systems with per-value QoS support. Hacked runtime hooks that tweak the garbage collection or memory management for particular use cases. Homegrown monitoring stacks and instrumentation systems. External configuration management tools. Custom build tool integrations with Jira.
And the upgrade slogs... when the database engine needs to be updated, or a new version of the underlying web technology is released.
Simple to build and simple to add to are not the same thing as 'simple'.
But when you start adding business rules, and start transforming your data and moving it around, then your monolith will become too complex and often too expensive to run. Lots of moving parts tightly coupled together, long-running transaction wrapping multiple joined tables etc. Rolling out new features will become very challenging as well.
Something like event sourcing is more complex to set upfront than a monolith, but at least it offers the way to add scale and features later without creating a combinatorial explosion...
Not saying it needs to be spaghetti all over the code mind you. Just that it's easier to have a module within the monolith rather than a dedicated service.
Building effective monoliths absolutely demands competent DB experts, schema designers & developers. The problem that microservices solved was the sparsity of this sort of talent when demand overshot supply by orders of magnitude.
(In a way the monoloth vs microservices debates are echoes of the famous impedance mismatch between object graph runtimes and relational tables and DBMSs.)
Stackoverflow runs on a couple of (beefy) machines only.
(Meaning generally that performance-wise, they clean the clock of any microservice-based system. Serializing a structure, shipping it over a network with compression and encryption, unserializing it on the other end, performing some operation, serializing the result, shipping it over the network with compression and encryption, deserializing the result, and possibly having to link it back up to internal data structures finds it hard to compete with "the data is already in L1, go nuts".)
I've even successfully extracted microservices from them when that became advantageous, and it was a matter of hours, not months, because I've learned some pretty solid design patterns for that.
If you can't design a well-structured monolith, you even more so can't design a microservice architecture.
It's not wise to try to learn too many lessons about what is good and bad from undisciplined, chaotic code bases. Chaos can be imposed on top of any nominal architecture. Chaotic microservices is not any more fun than a chaotic monolith, it's just unfun in a different way. The relevant comparison is a well-structured monolith versus a well-structured microservice architecture and that's a much more nuanced question.
That's why I mentioned Event Sourcing pattern, not "microservices". Think of a single event log as a source of truth where all the data goes, and many consumer processes working in parallel alongside, picking only those events (and the embedded data) that concern them, reacting to them, then passing it on not knowing what happens later. Low coupled small self-sufficient components that you can keep on adding one next to another, without increasing the complexity of the overall system.
Maybe Event Sourcing/CQRS can be called "microservices done right", but that's definitely not those microservices (micro-monoliths?) everyone is talking about.
For instance, an immutable event log is illegal in many cases (PII). So you have to either do compaction on the log or use an outside mutable store.
Another issue is code evolution: if you change your event processing logic at runtime, you get a different state if you replay it. Maybe some users or orders will not be created at all. How you deal with that? Prevent it with tooling/testing or generate new events for internal actions?
Also, all the derived state is eventually consistent (so far so good) but for non-toy apps you absolutely need to use derived state to process events, which naively breaks determinism (now your event processing depends on the cursor of the derived state).
Check out Rama[1]. They’re solving this problem, and it’s super interesting but again let’s not fool ourselves – we’re far from mature and boring now.
Something like it could hopefully become boring in the future. Many of these features could probably be simplified or skipped entirely in later iterations of these patterns.
Absolutely NONE of that has to happen if you structure your project/code well.