Working on complex systems: What I learned working at Google
thecoder.cafe
thecoder.cafe
Yes, they are Google; yes, they have a great pool of talent around; yes, they do a lot of hard stuff; but most of the time when I read those articles, I miss those kinds of distinctions.
Not lowballing the guys at Google, they do amazing stuff, but in some domains of domain/accidental/environmental complexity (e.g. sea logistics, manufacturing, industry, etc.) where most of the time you do not have the data, I believe that they are way more complex/harder than most of the problems that the ones that they deal with.
In reality, lazy domain owners layered on processes, meetings, documents, and multiple approvals until it took 6 months to change the text on a button, ugh
Most smaller places don’t have the bandwidth and many larger ones don’t have the desire.
I’m not sure if that makes up for bugs potentially introduced in the refactors, though.
Look, sometimes you may have good reasons for why a test is impractical. You are allowed to push back, or look for a different reviewer. There's a hundred thousand people in the firm, you should be able to find one or two that will let you submit literally anything that compiles.
But most of the time, the reviewer is giving you advice that you should take.
If there is actually only one owner that can approve changes to the package, there's something really weird and unusual about that project setup, or it's someone's internal hobby project that they wrote five years ago and semi-maintain, in which case, I have to wonder why you submitting one-liner changes to it is all that important.
We're all adults, we all work together, we can all work this out. If someone absolutely insists on being an asshole, escalate. It's why you have a manager, and why they have a manager.
My experience is that very few people are unreasonable assholes.
There's always plenty of organizational, vision, strategy, and execution problem in any billion-dollar company, but 'people are unreasonable in code reviews' is not one I'd put in the top 10. It might be something that ruins your day once or twice, but that doesn't make it systemic.
That's doubling down on time spent on contributing back. It's usually cheaper to workaround the issue once you notice it'll be way harder than it should be (not hard at all).
I've allowed people to build a whole obstacle course one time. Decades later it stil has me randomly burst out in laughter. It's like hoarding technical debt until nothing in the code base makes sense or even looks familiar. You just don't do that...
They're acting as selfish demanding you do something for them, as you are for refusing.
All because it may break someone. Even when I presented a real defect based on docs/comments and fixed it. You'd think that if they truly cared about breakages they'd already have some tests for it from where I can easily start.
It's great that you found a bug and fixed it.
The problem is, how do you know that there are no other regressions?
Code is a liability. Once you check it in, the team that owns it is responsible for it. Untested code is a liability of unknown scope. It's quite understandable why they don't want to accept someone's contributions, when the contributor isn't the one who will ultimately be dealing with any of the consequences. If you think they are being mean and lazy, imagine if the tables were reversed.
I don't accept puppies or elephants as gifts for similar reasons.
It's unfortunate that existing test coverage sucks. In this case, the best way forward should be for the team in question to improve coverage, and for you to then submit your fix + a test for it. And if they don't have budget to do this, then that sucks, but that's their call to make, and that's a signal that the project in question is abandonware.
And it's fine for a large company to have a bunch of abandonware. If it works, and produces value, the optimal amount of ongoing development effort to invest into a piece of software may, depending on the circumstances, be near-zero.
I've been on every side of this conundrum. Not being allowed to improve/fix stuff is super frustrating, very demoralizing.
I can quickly recall 3 separate projects where we weren't allowed to fix stuff on new products that hadn't shipped yet (not even beta). (Oops, just thought of 2 and 1/2 more projects.)
It's hard to not treat that as beligerent, spiteful, gatekeeping, control issues. For me, half the time it absolutely was.
I've also managed legacy code bases that we dare not accidentally breathe on. Somewhere between archaeology and necromancy.
For me, when I'm in charge, project (engineering) management comes down to mitigating risk, usually by driving down the cost of change. I'm sure I'm preaching to the choir.
Alas, it's been a long time since I've been on a project that had anything resembling proper PMI, QA, Test, etc. So there's really no process or strategy to mitigate risks.
IIRC, the contemporary deliberate rejection of process is called something like the "Argyle" Methodology. Or maybe it's "Agile". Or "Argh-gargle". I don't quite remember. I can dig up those books and links, if anyone is pervously curious.
Well. Good luck.
Interesting. As a consultant for the most of the last 25 years, my experience is the domain owners are typically invested and have strong opinions on the things that impact their jobs.
Executive leadership, on the other hand, doesn't want to actually know the issues and eyes glaze over as they look at their watches because they have a tee time.
This is only partially down to the impossibility of having every staff member on a project be A++ players.
There is coordination RISK not just coordination overhead. Think planning a 2 week trip with your spouse with multiple planes/trains/hotels, museum/exhibit ticket bookings, meal reservations, etc. Inevitably something gets misunderstood/miscommunicated between the two of you and therefore mis-implemented.
Now add more communication nodes to the graph and watch the error rate explode.
Those jobs also include things like management and product design, and so the coordination risk is reflected in the 5% chance that the manager drops the ball on communication. (As a manager, I suspect that chance is significantly more than 5% and that's why overall success rates are even lower.)
And that under-estimation compounds to make the top level 60% much higher than it should be.
A 7.5% rate takes top-level success odds below 50% - 46%. A not unrealistic 10% takes the top level down to 35%.
Etc.
Every project carries with it three possibilities: that of success, where the company makes money, that of failure, where the company does not, and that of a "critical failure", where the project goes so wrong that it results in a major lawsuit, regulatory fine or PR disaster that costs the company more than the project was ever expected to make.
If you're a startup, the worst that can happen to your company is the value going to 0. From an investor's perspective, there's not much of a difference between burning all the money ($10m) and not finding product-market-fit (normal failure), or your company getting sued for $3b and going bankrupt (critical failure). The result is the same, the investment is lost. For a large corporation, a $3b lawsuit is far more costly than sinking $10m into a failed project.
You can trade off these three possibilities against each other. Maybe forcing each release through an arduous checklist of legal review or internationalization and accessibility testing decreases success rates by 10%, but moves the "critical failure rate" from 1% to 0.5%. From a startup's perspective, this is a bad tradeoff, but if you're a barely-profitable R&D project at big co, the checklist is the right call to make.
This problem is independent from all the other causes to which bureaucracy is usually attributed, like the number of layers of management, internal culture, or "organizational scar tissue." Just from a legal and brand safety perspective, the bigger your org, the more bureaucracy makes sense, no matter how efficient you can get your org to be.
Just like anything else in life you want to look at the present value and then get insurance for huge risks.
That said agree that a startup can take more risk but I don't think that is the major factor explaining why larger companies tend to be process heavy and slower.
Other things in a similar category are:
- negative media attention (media scrutiny increases proportionally to organization size)
- doing something that upsets an influential group and may have consequences for the rest of your business (think how big the outrage would have been if Apple, Google or Microsoft tried making an "Uber" app before Uber existed)
- bringing down the service which is being worked on, potentially breaking SLAs
- failing to meet customer / legal commitments, particularly in regards to internationalization, accessibility etc.
- security incidents, which are presumably a bigger deal, as your project is connected to the rest of your infrastructure
- getting cancelled online, which causes employees (unrelated to the project) to quit
- natural, random and serious consequences that result from the fact that your project needs the company to hire additional employees. E.g. there's a certain number of people willing to commit sexual assault or financial fraud in the population, and the more people you hire, the more likely it is that you get one of them.
Process, complexity, inefficiency is just something that happens for big companies and big software. Things slow down and then there's a negative feedback loop and things just go down hill. Innovators dilemma sort of stuff.
I wonder if a related intensifier is that as a company grows larger it tends to follow the letter of the law rather than the spirit of the law, which results in less buffer space between the behavior and the law (and hence higher lawsuit risk).
Google has all sorts of great infra projects that simplify complex domains. Those are solved problems now, so nobody notices it.
The existence of incidental complexity isn’t evidence that the counter factual is less complexity.
No greater Scott’s man can tell you that the reality is surprisingly complex, and sometimes you have resources to organize and fight them, and sometimes you use those resources wiser than the other group of people, and can share the lessons. Sometimes, you just have no idea if your lesson is even useful. Let’s judge the story on its merits and learn what we can from it.
Arbitrarily, 95% of the time the issues were people problems, not technical ones.
On the other hand, the good part of the job is solving complex technical problem with a team.
This qualify as complicated. Delving in complicated problems is mostly driven by business opportunity, always has limited scaling, and tend to be discarded by big players.
That doesn’t feel right.
Let me bring a non-trivial, concrete example—something mundane: “ePOD,” which refers to Electronic Proof of Delivery.
ePOD, in terms of technical implementation, can be complex to design for all logistics companies out there like Flexport, Amazon, DHL, UPS, and so on.
The implementation itself—e.g., the box with a signature open-drawing field and a "confirm" button—can be as complex as they want from a pure technical perspective.
Now comes, for me at least, the complex part: in some logistics companies, the ePOD adoption rate is circa 46%. In other words, in 54% of all deliveries, you do not have a real-time (not before 36–48 hours) way to know and track whether the person received the goods or not. Unsurprisingly, most of those are still done on paper. And we have:
- Truck drivers are often independent contractors.
- Rural or low-tech regions lack infrastructure.
- Incentive structures don’t align.
- Digitization workflows involve physical paper handoffs, WhatsApp messages, or third-party scans.
So the real complexity isn't only "technical implementation of ePOD" but; "having the ePOD, how to maximize it's adoption/coverage with a lot uncertainty, fragmentation, and human unpredictability on the ground?".
That’s not just complicated, it’s complex 'cause we have: - Socio-technical constraints,
- Behavioral incentives,
- Operational logistics,
- Fragmented accountability,
- And incomplete or delayed data.
We went off the highly controlled scenario (arbitrarily bounded technical implementation) that could be considered complicated (if we want to be reductionist, as the OP has done), and now we’re navigating uncertainty and N amount of issues that can go wrong.
My take is that if your complex problem is only solvable by complex software (e.g. not a combination of simple small parts), and _cannot_ be reduced to simpler things, you are in the complex space.
Maybe it's too reductive, it's just my opinion, but it's a good way for me to determine predictability on ability to solve a problem with many unknown, at the engineering level. The dangerous blockers are in complex space, identifying them early is critical. Complicated stuff can be worked around and solved later.
Complicated on the other hand, involves the addition of one or more complicating factors beyond just "the problem is big". It's a qualitative thing, like maybe nobody has built adequate tools for the problem domain, or maybe you don't even know if the solution is possible until you've already invested quite a lot towards that solution. Or maybe you have to simultaneously put on this song and dance regarding story points and show continual progress even though you have not yet found a continuous path from where you are to your goal.
Climate change is both, doing your taxes is (typically) merely complex. As for complicated-but-not-complex, that's like realizing that you don't have your wallet after you've already ordered your food: qualitatively messy, quantitatively simple.
To put it differently, complicated is about the number of different domains you have to consider, complex is about--given some domain--how difficult the consideration in that domain are.
Perhaps the author's usage is common enough in certain audiences, but it's not consistent with how we discuss computational complexity. Which is a shame since they are talking about solving problems with computers.
“Simple Made Easy”: https://youtu.be/SxdOUGdseq4?si=H-1tyfL881NawCPA
1) scale of application variety (10k different apps with different needs and history)
2) scale of human capability (ingenuity), this scale starts from sub-zero and can go pretty high (but not guaranteed)
Google, where the impossible stuff is reduced to merely hard, and the easy stuff is raised to hard.
“I just want to store 5TiB somewhere”
“Ha! Did you book multiple bigtable cells”
Obviously, there are limits to how many pay bonuses you can give out and if it’s direct money or store credits.
Directly asking for a peer bonus’ is not very “googly” (and yes, this is a term they use- in case you needed evidence of Google being a bit cultish).
There are companies who help do this “as a service”; https://bonusly.com/
It never ceases to amaze me how (early) big tech embraced and even promoted things that would have been considered "career limiting" in traditional big corporations.
Since that process is invisible to those being measured you never know details (and shouldn't as long as management is sane, and if isn't this the least of your concerns), but its not ignored and in this way it helps keeping people motivated to generally do good work.
How so?
Obviously there are other professions that share some of these characteristics, like sales. Or if you narrow down a goal or task to "save us money".
Funny way to spell "unpaid extra work".
To prevent obvious abuse, you need to provide a rationale, the receiver's manager must approve and there's a limit to how many you can dish out per quarter.
It is a social engineering technique to exploit more work without increasing wages. Just like "Employee of the Month" or a "Pizza Party."
Company I work for does this with gift cards as rewards. I was reprimanded because I sent an email to HR that this " gift" is as useful as a wet rage in the rain. I don't eat at restaurants that are franchises or have a ticker on Wall Street. Prefer local brick and mortar over Walmart and will never financial support Amazon.
If you want to truly honor my accomplishments, give me a raise or more PTO. Anything else is futile. That gift card to Walmart has 0 value towards a quality purchase like a RADAR or LiDAR development kit to learn more or such.
It technically requires manager approval but it's kind of a faux pas for a manager to deny one unless it's a duplicate.
Do not miss
This was posted in my front office when I started my company over 30 years ago.
It was a no-brainer, same thing I was doing for my employer beforehand. Experimentation.
By the author's distinction in the terminology, if you consider the complexity relative to the complications in something like Google technology, it is on a different scale compared to the absolute chaos relative to the mere remaining complexity when you apply it to natural science.
I learned how to do what I do directly from people who did it in World War II.
And that was when I was over 40 years younger, plus I'm not done yet. Still carrying the baton in the industrial environment where the institutions have a pseudo-military style hierarchy and bureaucracy. Which I'm very comfortable working around ;)
Well, the army is a massive mainstream corp.
There are always some things that corps don't handle very well, but generals don't always care, if they have overwhelming force to apply, lots of different kinds of objectives can be overcome.
Teamwork, planning, military-style discipline & chain-of-command/org-chart, strength in numbers, all elements which are hallmarks of effective armies over the centuries.
The engineers are an elite team among them. Traditionally like the technology arm, engaged to leverage the massive resources even more effectively.
The bigger the objective, the stronger these elements will be brought to bear.
Even in an unopposed maneuver, steam-rolling all easily recognized obstacles more and more effectively as they up the ante, at the same time bigger and bigger unscoped problems accumulate which are exactly the kind that can not be solved with teamwork and planning (since these are often completely forbidden). When there must be extreme individual ability far beyond that, and it must emanate from the top decision-maker or have "equivalent" access to the top individual decision-maker. IOW might as well not even be "in" the org chart since it's just a few individuals directly attached to the top square, nobody's working for further promotions or recognition beyond that point.
When military discipline in practice is simply not enough discipline, and not exactly the kind that's needed by a long shot.
That's why even in the military there are a few Navy Seals here and there, because sometimes there are serious problems that are the kind of impossible that a whole army cannot solve ;)
They have a unique and distinctive experience, but it usually isn't what you expect. It is rare to encounter someone from Google that actually built something of significance, and those that have are always at the staff+ level and had been there 10+ years.
If I were to make another generalization, the [g|x]ooglers that worked on relatively "small" projects are often the most interesting, as they had resources to build something from the ground up and do attempt some really interesting projects.
"We can do it!" confidence can be mostly great. (Though you might have to allow for the possibility of failure.)
What I don't have a perfect rule for is how to avoid that twisting into arrogance and exceptionalism.
Like, "My theory is correct, so I can falsify this experiment."
Or "I have so much career potential, it's to everyone's advantage for me to cheat to advance."
Or "Of course we'll do the right thing with grabbing this unchecked power, since we're morally superior."
Or "We're better than those other people, and they should be exterminated."
Maybe part of the solution is to respect the power of will, effort, perseverance, processes, etc., but to be concerned when people don't also respect the power and truth of humility, and start thinking of individual/group selves as innately superior?
Actually, I've found this is a constant in life, whatever you achieve, you end up in a situation where you're pretty average among your peers. You may feel proud to get into Google for a few months, and then you're quickly humbled down.
(Also, to be clear about my examples: I don't think Google is fabricating their dissertation research, nor do I think Google is genocidal.)
If you're suggesting there's a lot of humility, yet "But we can do it!" still works, that's great, and I'd be interested in more nuances of that.
Clicking on "Only Necessary" causes the cookie policy agreement to reappear.
GDPR banner only to be used in EU, with conditional of only accepting non-essential cookies, and the engineer or QA is based in the U.S.
As a side note, as someone that lives in the EU my pattern of usage here is:
- choose only non-essential, but if not a presented option then
- reject all cookies, but if no reject all available then
- switch the reader mode (or hide distracting items), or if not possible then
- close tab
I’m getting much more aggressive when dealing with cookie banners dark patterns. I will not a third third party advertising cookies as much as possible and support websites that allow me an easy way to opt out of them.
One myth is that complex systems are inherently bad. Armed forces are incredibly complex. That's why it can take 10 or more rear echelon staff to support one fighting soldier. Supply chain logistics and materiel is complex. Middle ages wars stopped when gunpowder supplies ran out.
Another myth is that simple systems are always better and remain simple. They can be, yes. After all, DNA exists. But some beautiful things demand complexity built up from simple things. We still don't entirely understand how DNA and environment combine. Much is hidden in this simple system.
I do believe one programming language might be a rational simplification. If you exclude all the DSL which people implement to tune it.
Ukraine would be conquered by russia rather quickly if russians weren't so hilariously incompetent in these complex tasks, and war logistics being the king of them. Remember that 64km queue of heavy machinery [1] just sitting still? This was 2022, and we talk about fuel and food, the basics of logistics support.
The arquebus is the first mass gunpowder weapon, and doesn't see large scale use until around the 1480s at the very, very tail end of the Middle Ages (the exact end date people use varies based on topic and region, but 1500 is a good, round date for the end).
In Medieval armies, your limiting factor is generally that food is being provided by ransacking the local area for food and that a decent portion of your army is made up of farmers who need to be back home in the harvest season. A highly competent army might be able to procure food without acting as a plague on all the local farmlands, but most Medieval states lacked sufficient state capacity to manage that (in Europe, essentially only the Byzantines could do that).
https://cloud.google.com/storage/docs/hosting-static-website + pick your favorite OSS CMS
"This is one possible characteristic of complex systems: they behave in ways that can hardly be predicted just by looking at their parts, making them harder to debug and manage."
https://en.wikipedia.org/wiki/Complex_system
although I understood the key part of a system being complex (as opposed to complicated) is having a large number of types of interaction. So a system with a large number of parts is not enough, those parts have to interact in a number of different ways for the system to exhibit emergent effects.
Something like that. I remember reading a lot of books about this kind of thing a while ago :)
Unless your problem comes from something side effects on a computer that can’t be modeled mathematically there is nothing technically stopping you from modeling the problem as mathematical problem then solving that problem via mathematics.
Like the output of the LLM can’t be modeled. We literally do not understand it. Are the problems faced by the SRE exactly the same? You give a system an input of B and you can’t predict the output of A mathematically? It doesn’t even have to be a single equation. A simulation can do it.
The core problem is building a high enough fidelity model to simulate enough of the real world to make the simulation actually useful. As soon as you have some system feedback loops, the complexity of building a useful model skyrockets.
Even in “pure” functions, the supporting infrastructure can be hard to simulate and critical in affecting the outputs.
Even doing something simple like adding two numbers requires an unimaginable amount of hidden complexity under the hood. It is almost impossible for these things to not have second-order effects and emergent behaviour under enough scale.
However, all this amazing stuff in the service of .. posting ads ?
Dont be reductive.
Both python and rust (for instance) are both turing complete, and equally capable
Whatever you're working on, your project is not likely to be at Google's scale and very unlikely to be a "complex system".
Just because your project might not be at Google's scale doesn't mean it is therefore also not complex [^1]
Example: I'd say plenty of games fit the author's definition of "complex systems". Even the well-engineered ones (and even some which could fit on a floppy disc)
[1]: https://en.m.wikipedia.org/wiki/Affirming_the_consequent
A. The same reason Amazon had/has such a hard time.
B. Google lacking the same persistence of Amazon (Consider all the products that are killed)
C. Google's hiring process. (They organizationlly do not know how to hire specialists)
Yah, like the Stadia, Google's streaming gaming console thing. They even had a first party game development division for it. So exactly what OP was wondering about.
Fundamentally, and ironically, Google likes to offload complexity on to everyone else in their ecosystems, and they got so used to people being willing to jump through hoops to do this for search ads/SEO they are very confused when faced with a more competitive environment.
One reason Google can't make games is they can't conceive of a simple enough platform on which to design and develop one. It would be a far too adventurous constantly moving target of wildly different specifications, and they would insist you support all possible permutations of everything from the start. There are reasons people like targeting games consoles, as it lets you focus on the important bits first.
And Apple :)
The two mobile duopolists don't understand gaming.
7 is the number of different AWS services he managed to use for just a prototype.
We thanked him and rewrote the thing.
Even a project that's like 15k lines of code would benefit from a conscious effort to fight against complexity.