Software Engineering Insights from 10 Years at Google
addyosmani.com
addyosmani.com
It may seem harsh, but the industry should stop elevating this kind of pablum.
On the other hand we have a bunch of people spending too much time on hacker news, often with all the insight of someone who spent 5 minutes scanning a wikipedia article. I'd worry a lot more about the long term effects of elevating this self congratulatory content, but where's this content cop then?
> It may seem harsh
Honestly it seems like it's trying too hard. Call it feel good vagueries and move on.
Addy is pretty well known in the frontend community. He's published a bunch of books, has hosted the Totally tooling tips podcast, gave a bunch of talks on performance, and has appeared in popular frontend publications, such as the Smashing Magazine.
Anyways, Addy Osmani did contribute or did talked a lot about Frontend engineering development, especially wrt to Google Chrome. Whenever I think of Lighthouse, or try to debug or get my team to debug with Google Inspector, I remember him. I have read a lot of his articles, public tweets (I think) where I have exclaimed "oh! That's how", "ah! that is cool."
Signed, another googler who couldn't answer any of those questions despite half a decade.
Google publicly cancels a lot of products. It has a pretty bad long term product strategy. But part of that is because it has like 100000 engineers. Most people aren’t experiencing political problems.
Dealing with political problems, technical debt, and changing product requirements are all core parts of navigating the software industry.
If you need to go on tour with your rock band you don't go to a Music conservatory and start interviewing musicians based on their knowledge of scales. You give them a Guitar, or Drums or Bass and you ask them to play something. You will know right away if they are the Guitar player or Drummer you are looking for.
That is why Software Engineering hiring is still broken, and why companies need to keep buying other companies to acquire successful software products. And that is why nobody cracked yet the challenge of successful software development.
Some of the best Software Developers are self-taught, and that include in this category, the ones with a different type of degree, that moved into Software Development.
You don't go to a school and come out an Eric Clapton or a Jimi Hendrix. You either have it or not. It's the same for Software Engineers and Software Development. No amount of schooling will help you if "don't have it" and no lack of schooling will stop you from developing something and share it with the world.
The equivalent of this in software engineering is to give them a coding problem and asking them to write code to solve it and test it, which does happen in interviews.
The theoretical concepts are also often discussed in technical interviews. This is useful as it proves the ability to have a discussion with other engineers without getting bogged down in fundamental misunderstandings. If someone doesn't know what an array or a hash table is, that's going to generate difficulties for collaboration.
I think an orchestra is a better analogy than a small music band, when it comes to big companies. You wouldn't hire a violinist who can't read music or understand all the theory required to interpret written music.
Initially, you say: you need the core knowledge of algorithms and data structures (like the circle of fifths and harmony) to do sophisticated improvisation.. so you're advocating for the importance of having a strong technical background. OK, fair.
But I don't get your analogy to a rock band going on tour, and they either have it or they don't? What is the software analogy to going on tour? Is this working in a startup, when you just need results rather than finesse? How does this tie back to your initial point about jazz musicians needing a core base of knowledge?
You also bring up the myth of the self-taught, scrappy musician (or programmer) who, against all odds,'has it' and is able to exceed the abilities of others who are the hardest-working and most studious.
I find this concept pretty toxic. The Eric Claptons and the Jimi Hendrixs of the world are a rarity, a complete exception. Although often cited because they are incredibly visible, compared to the hundreds of thousand if not millions of others who either developed skills and capabilities through education and/or self-taught effort. Often without 'having it'. I think it can be pretty discouraging when everyone holds themselves up to ideal of Claptons and Hendrixs.. rather than considering there might be multiple, varied and nuanced paths to being an excellent engineer, or musician.
What I think is toxic is to make many believe they can get a Computer Science education and become a professional Software Developer. The same way finishing a five year Music degree might still mean somebody is terrible at the Piano.
You can construct various simple problems (that require only loops and arrays) that can easily be mapped to real world problems and see how well the programmers perform (problems: that exploit the fact that input is sorted, linear search for min/max, collect data -> loopy state transition -> output state).
A fictional scenario that you can map to a real software problem, there's 7 cities, N people (program input) in city 0. 6 buses, one goes from city 0 to city 1, other from 1 to 2, ... last from 5 to 6.
Bus has a limited capacity (6 additional program inputs). Everyone needs to be at city 6. Time starts at 0. One time tick: all buses go from city X to city (X+1), unload people at city (X+1), and then go back to city X. What's the minimum time it takes for everyone to be at city 6?
There's 7 inputs (N people and bus capacities) and there's multiple programs one can write to solve the problem. There's no math tricks, no data structures.
You can write a program whose time complexity depends on the size of the inputs O(N), or just on the number of inputs O(7), or maybe something worse or in-between.
I have no idea if there is a way for one to teach/learn how to write the simplest program as soon as possible, but some will find it in 1 minute, some in 60 minutes. The amount of time it takes might be irrelevant and the only thing you're looking for is the ability to eventually think simple enough. And the thinking bit is not the only filter. The person might struggle to use the computer (but uses it well enough for these simple problems), might have no idea about the machine that runs the program, etc.
Let's say I am interviewing, for their corresponding roles, one of the Aerospace Engineers responsible for the design of the latest Airbus, or a specialized heart surgeon with 10 years of experience, or a Civil Engineer that lead 5 year projects building bridges crossed daily by thousands of persons.
What kind of look would they give me....If I start throwing them the type of mental puzzles like that, that you typically find on the back pages of newspapers on dentists waiting rooms:-)
Programming as a craft, generally, does not have a "great filter" like medicine, civil engineering or aerospace. The longer one survives in these fields, there's a really small chance of surviving out of luck and not out of experience.
If you look at carpentry instead, the mental part of the craft is also one thing you can test. For example, you might provide a limited set of tools (like arrays and loops) and require a solution to a carpentry problem. The mental part is not tested by puzzles (by puzzle I assume there's a set of tricks you need to know, or in the case of programming, a particular algorithm), but by problems. Problems might be more concrete for carpentry.
The problem above with buses can be reframed into whatever the area of programming expertise is for a candidate. There's a simple underlying structure in the problem that can appear in web dev, networking, databases, that maps directly to that bus problem. If you cannot figure out the buses, then how will figuring it out for something else be simpler? The solution is just loops and arrays, there's no fancy algorithms, there's no fancy math, it's pure art of programming.
Like I said, it's only one part that you're testing, which is thinking/problem solving. It can also be affected by personal anxieties or whatever. Failure at a particular problem does not mean much. But I would not say that these kinds of coding tests break hiring.
I mean, I hear people are being asked to pick if they're more hardworking or creative, or to describe where they see themselves in 10 years, or some other similar abstract silly stuff. I'd much rather someone asks me about buses.
You are working as an Interviewer for a FAANG and the six candidates today for a SWE position are:
- Linus Torvalds
- John Carmack
- Fabrice Bellard
- Guido van Rossum
- James Gosling
- Donald Knuth
What "puzzle" are you proposing?
FizzBuzz is not a puzzle solvable only with an obscure trick, and easily maps to real world, the buses problem too, and I can list more problems that are solved with 1-10 lines of code but the initial attempt used 20-50 because the structure of the problem was not fully exploited, unnecessary passes over the data, too much intermediate state etc. (all of the listed bloat appears in real world all of the time)
There are no tricks, no maths, no algorithms, no data structures involved. Your ability to pattern match these little bits should grow with your experience.
And why are so many competitive programmers,(not all) terrible Software Engineers?
https://kislayverma.com/organizations/competitive-programmin...
https://www.freecodecamp.org/news/mythbusting-competitive-pr...
Of course you're going to have to spend weeks on LeetCode if interview questions target knowledge of dynamic programming, heaps etc.
You're preaching to the choir.
My point was that FizzBuzz might test individuals to see if they can code, but there are "harder" problems that happen on a daily basis, and the amount of years you put in will affect your ability to pattern match.
Competitive programming captures a part of the essence of programming, you're writing minimal, fast and efficient programs. You improve your pattern matching to minimize the amount of intermediate state, unnecessary loops etc. After some point, competitive nature pushes you to specialize, and then it probably stops making sense if you want to make programming your craft.
Isn't that off putting and isn't that ingraining distrust from the very beginning?
Are open form questions for which the candidate can give an honest answer or that start a discussion better, or do you find them similar to a simple coding problem? What if during discussion I probe for more details about one particular thing?
Just to be clear, I agree that coding questions can be ridiculous. These do appear in some interviews. Questions related to quicksort, radix sort, counting sort, heaps, Dijkstra, 0-1 bfs, dfs, iterative deepening, prime testing or integer factoring, or convoluted problems of counting the many ways of tiling a n x m grid with 2x1 and 1x1 tiles, can all be found in the wild.
My buses problem is completely different and simpler.
As I read through the article, I couldn’t figure out if it was me “hating” on the author or if it really read like a 99 cent store hallmark card.
Glad to see I wasn’t the only one who thought the latter.
Even if you can’t find value in what was written just take it as a lesson in what not to do.
you wrote this. about someone else's writing
So no, I don’t have great advice for the author, but failure on that magnitude is obvious.
Honestly I think the better example would be from a tiny company that can't afford to do things wrong.
If you read what the majority of them recommend, its similar to the banker's path in the early 2000s. Graduate from Ivy League, spent 3 years at FANG, then 2 years at Ivy League MBA, then back to FANG or join a startup for a few years so you can get a title bump when you do go back to FANG. The type of people that join Google nowadays aren't the type of people that were early employees. They have a very standard charted course to their careers.
Google has reach maturity stage of the company. They have leeway to screw up like what Microsoft did with Windows and Office in the early 2000s. Will they have the ability to push out something completely new and amazing like Gmail or Google Maps (although Maps was a clone of Mapquest, they executed better), where the end users aren't developers themselves? I highly doubt it with an organization as big. There will be a lot of politics involved to get anything done. They have lost the talent to innovate.
What's stopping you?
I haven't found a problem where I have any insight on how to solve
What Google seems to do really well is building robust systems at a scale that pushes the limits of what humanity is capable of. If you collect anecdotes from engineers who've worked on scaling systems at the small number of companies at this scale, there seems to be general agreement that Google is "the best". My understanding from these anecdotes is that Google core services are "on fire" less than everywhere else and require less maintenance.
But that's a pretty narrow definition of software engineering. AWS, by all accounts, is much better at productizing something useful to the world. I've heard that the reason Microsoft lags Google in stability could plausibly be accounted for by the fact that they care so much more about maintaining backwards compatibility (which Google very clearly does not care about).
All that said, I don't think its "best at software engineering" is a terribly useful category, better to say they are plausibly "best at a very specific and very difficult kind of software engineering".
Whether core services are "on fire" depends on which team you're on. Some core services are pretty much "on fire" all the time, with high pager loads for the SRE teams supporting them. Some core services are rock solid, and the SRE teams conduct exercises where they turn the service off just to remind people that the uptime SLO is not 100%.
Same goes for code quality. Some teams have code that you'd just love to print out and put the code listings on your desk. Some teams have code that doesn't make sense and nobody understands.
Some teams are highly functional, some teams are exceptionally dysfunctional.
Can you give some examples of things Google did which pushed the limits of what humanity was capable of?
Or, how about a video distribution network that serves more than 600,000 hours of video every minute, and manages to do that such that, every time service deteriorates on my 4G connection in a poor country, it's the last website (even considering text only websites) to stop working?
As for YouTube, it was bought, not created by Google, wasn't it?
As for youtube, serving video is not the impressive part. It's the scale at which they do it. And my understanding is that the scale has increased exponentially since the 2006 acquisition.
Depends what you mean by "better at software engineering".
Conversely, simply having more market share doesn't mean AWS services are well engineered and the engineers are smarter (they might be, but you cannot make this claim).
There're different ways to measure "smart". My view is that in the cloud computing business there are people smarter than them (at least in terms of market share) so it's perfectly fine that Google Cloud engineers learn from their competitors in terms of engineering, marketing, etc.
What could have been... [1] Yes, I'm still bitter about it after all those years.
My impression is that this is a result of a policy that google3 never touches a developer machine, which of course means you can't disconnect and work from a log cabin in the woods. Really sounds like bending yourself around an antipattern to be honest.
This is becoming an increasingly remote (hah) scenario where doing development over web-based IDEs and SSH becomes a problem. For the incredibly few people who do this sort of thing regularly, you could get satellite internet reimbursed. There is even a story of a SRE who used to work from the top of mountains on occasion because he hiked so much.
I wouldn't call them "evolving" anything software as of late. Their hard science departments seem good though, but these aren't software engineering
There are a lot of projects that look easy at a POC stage and are very different at million of users, millions of tps, millions of TB or full featured
Spanner seems a prototypical Google dev product though; it's only invented to serve Google. I don't think any of their solutions is the de facto choice in any tech stack.
It was invented to serve Google (really: the people who had outgrown bigtable's limited featureset) but is widely applicable.
80% of software engineering is not about pushing but maintaining. I'm not the one who praises Google (far from that), but using "pushing anything relevant" as a metric is not correct imho.
I understand there are many reasons for this (though some of those originate with Google itself), but that's rather the point. The whole reason Google became a thing is because most search engines 20 years ago were full of results that were "hyper-monetized, full of spam, and frequently just not especially great." Then Google came and offered a product that effectively solved this, and quickly became a giant because of it.
Maintaining, let alone evolving, means continuing to work at least as well in the face of changing conditions.
Looking forward to reading it when you publish the revised version, though I'm unsure how I would know. (I guess I should check and see if you have RSS)
I wish the civility rules were enforced here a bit more stringently sometimes.
It is sad to see that more often than not, the top comments on any article are overly negative or even vitriolic and any interesting discussion languishes deeper in the article.
I read your draft, and it is full of excellent advice. Some of it being which I also find very important, some which gives me food for thought and some which I am happy to be reminded of again. I look forward to your finished post.
What I noticed from the draft: There's nothing obvious about Google per se, in terms of the material. It could be company X you've been working at. Also, insights cumulatively cover junior dev, senior dev, and some manager experiences. So, audience for this particular piece was not clear to me.
I am surmising this piece complements your "Efficient Engineering" work-in-progress. I'll be curious to read that too.
Similarly, it'd likely be possible to learn many of the same findings at other companies -- Google (and the author) have been fortunate to encounter lots of tricky scaling and human issues in technology, and to evaluate potential strategies to deal with those, and it's good to see that knowledge shared.
My favorites from the list, that I did not see other companies (specifically startups) do at all:
> A single large release may be divided into a series of lower-risk well-understood rollouts.
> Design documentation should not be an afterthought but an integral part of the software engineering process.
> Coordinate reviews for the design doc and compare the design as it evolves with the original doc to verify that all the relevant constraints are being addressed.
It might suck to spend 4hr writing a doc and 1hr*(N engineers) reviewing it but getting everyone aligned, and writing down why you did something and what was considered but ignored, is one of the super powers people have. At larger companies this process, when done informally, can be done in under a day and save months of headaches down the line. Especially so when the reviewer leaves comments like "can we break X up into milestone A, B, and C instead of rolling our a single piece".
The moment this process becomes codified into a formal doc that requires approvals and reviews across teams, is the moment this ends up becoming a far larger part of the work than it should. Making it a required aspect of the job dramatically slows down the work and makes the doc act as a gatekeeper to technical work.
Maybe our releases are too big but our technical specs take much longer to write and even longer to coordinate stakeholders when conflicts do come up.
Can people really crank these docs out in 4 hours?
> The best software is built by engineers who have empathy for their users.
I’m not sure where all the criticism here comes from but those alone are some of the most valuable insights any developer can obtain.
The fact that Google, in my opinion, is as far from those insights as a company can possibly come is in my eyes unrelated to the lessons that the author has learned working for Google. I think there is way too much anger in this thread, and I’m surprised to see it here on HN. I’ve personally moved all my services away from Google, including the paid ones like gsuite, but this article doesn’t seem to warrant any of the harsh criticism here based on its content.
This post is the perfect example. It lacks an original premise, a unique sense of language, or an understanding of the relationship between incident and narrative. Most of its core ideas are hokum: 'I think it was the immortal Steve Jobs who said "don't worry about getting wet, just dance in the rain"'.
But Google isn't a writing company. You could work there for a hundred years and never gain the ability to write anything worth reading. There's no evidence that the experience produces unique literary insight. Instead, it seems to create exactly this kind of sludgy, imprecise advice. All of which is unintentionally revealing.
Good luck with the final article!
I have a counter example. Founder of a very successful startup (early 2000's) once told me that they first built the technology (adding tracing to Java apps at run-time) just because it was interesting, without any clue if it can be useful to someone. Later they showed it to everyone and they got their first customer who agreed to try it (it required modification to JRE).
the last time Material worked flawlessly with Bazel outside Google was arguably MDL, if you could even count that as working.
i just don't understand why Google isn't investing in OSS the way it used to. why it cannot seem to do the thing that should be easiest: to make its own software work with itself. it seemingly cannot do this even in high-value categories it cares deeply about, like Cloud.
i loved MDL. thank you for writing it. but i wish you would write something that cuts to why it is 2022 and Google just announced Flutter integration with Firebase like we all don't know that absolutely should have been a basic staple feature from the get go.
The large pull quotes (which I’ll be honest I’m never a fan of) just interrupt me trying to figure out the layout.
So I’m not sure how things are organized.
I’m on my phone now, maybe it’s better on desktop. Reader mode didn’t really change anything so unfortunately that didn’t help.
The article sounds interesting.
Figures.