The maze is in the mouse: what ails Google
medium.com
medium.com
That said, I take immense issue with the characterization of the "heroism bad" sentiment at Google:
> There are documents that explicitly and proudly deride “heroism” and assert that not only should product teams not encourage “heroes”, they should actively dissuade them. If someone chooses to work twice as hard as is expected of them, they usually will be prevented from doing so because they have to work with others and doing so would force the others to work harder too.
As it's been explained to me, this is actually a principle primarily around avoiding inefficient hard work to keep a system healthy, more of a "work smart not hard" thing. Being the only person on the team who can keep prod healthy is not valuable when you could get hit by a bus tomorrow, and this behavior doesn't scale well. That's the gist of the argument and I think it is a good one. His criticism of people not taking enough risk is apt but that's not what this particular piece of culture is about.
What you’re saying is a virtue in peacetime and a sin in wartime. Precisely the argument the author is trying to make
I understand there are some exceptions: sometimes during outages/emergencies, you do want folks to fix issues ASAP regardless of how long they are up.
Definitely not. Heroism is nearly the opposite, in that it wants heroes to be significant, not the work.
Let's just level set for a second:
* Peer bonuses are a fixed $175 and spot bonuses are usually under $1000, and pretty much always under $5000.
* The coveted "feats of engineering award" (or whatever they changed the name to this time), which has the highest prestige factor of any of the internal awards, is $2000 cash or a trip somewhere.
* A senior SWE at Google makes about $300k. A Staff SWE makes $400-500k. An entry-level SWE makes $150-200k.
What this means is that the only REAL reward on offer (other than a token amount of cash and a serotonin-inducing certificate) is a promotion. Promotions almost always come when you launch products (actually, they changed the benchmark to "landing" products, which seems to always happen about 6-12 months after launch after the bugs are worked out).
Launches at Google are almost always significant efforts - either in terms of engineering around the company's arcane systems that are changing underneath you or in terms of getting through all of the process checks that ensure your service is "googly." Further, to go from a "launch" to a "landing" is mostly a political process: you have to convince enough of your management team that you have met some arbitrary milestone that indicates a stable product. As such, people are not too happy to take significant technical risks if they involve spending more than a few months on a project - if you want to get promoted, wasting time on something that might fail is bad. Why would you want to "land" a risky product when you can "land" something that is already on the roadmap?
Additionally, Google offers pretty much no promotion credit to engineers for "prospecting" - what I will call the process of identifying products that should go on the roadmap. That means that if you go out and do this sort of work, you are actively hindering your career: if you go prospecting, you are going to spend months not "landing" things. Conversely, prospecting seems to be a very good way to get peer and spot bonuses.
This is why Google provides no real benefits for taking technical risks. The only real reward is promotion, and promotions come mostly from following the roadmap, not from rocking the boat. The other awards and bonuses are basically toys by comparison.
And maybe well paid titles, but where one couldn't hire more people, one just got paid better? To mitigate over hiring?
**
Or what are your other thoughts, about how to solve the problems you noticed? ... In addition to:
> promotion credit to engineers for "prospecting"
interesting! it's precisely the point made at the end of The Caine Mutiny.
https://sre.google/sre-book/eliminating-toil/ is a good public example of this sentiment.
I can also see the value for a large organization in culturally enforcing interchangeability and replaceability of workers.
I always considered this bit of culture (and felt the message this way from coworkers) to be even more focused on on well-being of the would-be hero. It always comes alongside messages about how to deal with burnout or achieve a work-life balance that works for you. That's always been my preferred perspective on it, anyway.
Still, I'm with you in pushing back against the interpretation you quoted, which is at best a weak (and at worst toxically macho) understanding of real motivation behind this anti-"hero" sentiment. It's mildly absurd to say there's a culture of holding back hard workers in order to lighten the expectations for everyone else. Anyone who works "twice as hard as expected of them", broadly speaking, just gets recognized/rewarded/promoted until expectations and output match.
Edit: reorganized order of thoughts
You know what? Every project needed heroics just to get done on time because the place was profoundly fucked up. And each round of heroics left the code in poorer and poorer shape, demanding more heroics next time. Instead of doing a truly brave thing, like telling executives that a schedule was absurd and the team needed more time to do it right, people did the cowardly thing and said, "three bags full sir" to every absurd request.
And that was the right choice for promotion, because everybody higher up had succeeded by heroics.
as the old saying goes: "fix a production stopping bug after 30 seconds, you're an arsehole, fix it after 10 hours, a hero"
The problem with heroes is that they tend to be single people with a specialised and bottle-neck like ability. They step up because of a systematic failure.
Problems that actually require such efforts should be exceedingly rare. If they're not, then there's a deeper issue in the production process that desperately needs fixing.
You don't want developers (or anybody) to be working that hard on a regular basis. Burnout is real, and in the long run the practice will cost the developers and the company dearly.
Five Whys.
https://en.m.wikipedia.org/wiki/Five_whys#:~:text=Five%20why....
systemic?
The saying I know is "fix an outage => hero; prevent an outage => what are we even paying you for?
Heroes are a bottleneck in KTLO, but they are a necessity for innovation.
It's not a coincidence Google no longer innovates. They threw the baby out with the bathwater.
> There are documents that explicitly and proudly deride “heroism” and assert that not only should product teams not encourage “heroes”, they should actively dissuade them. If someone chooses to work twice as hard as is expected of them, they usually will be prevented from doing so because they have to work with others and doing so would force the others to work harder too. If someone says they can finish a project in a month, their manager will tell them to be realistic, pad it to four months, and tell the VP it is six months.
To me, that doesn't sound like avoiding all-nighters by lone wolfs so that everyone can support the system when it fails, it seems to me like it's preventing people from giving their all to create valuable products.
From my exposure to GCP from the enterprise side, this rings true. The folks we worked with were polite, but seemed to be asking us about our needs as a one-off / outside the normal process.
With credit to the teams we interacted with, they were helpful and acted on our feedback (things like "As a customer, I need a dashboard that tracks my GCP spending"), but it was apparent this wasn't something they frequently did with customers.
"The maze is in the rat" is a great metaphor.
Execs want certain things ("Realtime visibility into all projects!" or "Stability!") but underappreciate the organizational opportunity cost in requiring them.
Is it more important to have perfectly tracked projects or projects which ship 20% faster?
And if the former, we all understand you're training your people to be like forever, right?
Customers are those spending money posting ads on search. Frankly even they get crap customer support, but at least they are customers.
I'm no googler but it's clear "user support" (and "customer support") are not priorities at google, and that means other people _are_ priorities. So it's not exactly a shock to discover employees optimize to make other employees happy (measured) versus keeping users happy (not measured.)
I factor that in when I decide if I want to be more than a user, if I want to be a customer. (So far the answer is basically "no".) And, as a user, I understand I'm not a customer and Google doesn't care about me, or my gmail account... So I treat my gmail account with the importance it deserves.
So what though? This makes them happy, and they're all paid, so what does it matter what the outside world think or care?
There is a chasm between google cloud and microsoft azure. Azure sucks as a product, but they really, really, really want you as costumer. Damn, I get a weekly one-to-one with the guy which was part of the team that created their kubernetes offering, just because I needed someone with expertise in the matter to help us maximize the benefit or ours. NO EXTRA COST.
With Google I got a guy that knows Kubernetes less than me. I sometimes get embarrassed asking them questions because they don't know how to answer but must come up with something to answer because the account manager is in the call. It feels like their poor account managers take a uphill battle just to get someone to talk to us. I feel sorry for them.
Guess what happens? must deployments are in azure, even if the azure CLI sucks, the azure portal is a mess and azure products are inconsistent.
But Google's GKE ingress takes 20 minutes to update (there is a years old bug opened, nobody cares), vs Azure application gateway taking 30 seconds. Both being very bad for us, but I get to talk to the Microsoft the guy who worked on that integration and discussing with him the pros and cons of using ingress-nginx instead.
Google don't want us as costumers, they just want to brag about having a better cloud (they don't).
(as a PS, please microsoft your employees don't like to answer questions on stack overflow. But great to be able to get stackoverflow answers from the product managers).
As much as I hate personal relations, I frequently have to "herd cats" (contractors and cloud vendors). On the other hand, once in the while I get an absolute superstar and I work to keep them on my projects as long as possible.
When I was at Microsoft, the highest tier of support contract got you meetings with the lead developers of whatever product you found a pri0 bug in.
Found a way to crash the compiler and simple code changes weren't enough to work around it? A meeting with a member of the C++ compiler team would be arranged.
Microsoft has a history of customer focus, though I fear that was very much diluted when MS got rid of their dedicated test orgs.
Google brought in around $284B in 2022, Big customers are at least double digits, so for Google, one needs to spend at least ~ 30B/yr to be considered a big customer to them.
Let's assume that billions is actually 99B. So roughly 10B/yr. This is roughly 4% of their income in a year(~ 284B in 2022).
Big customers are at least double digit % of their income.
Their paid technical support (the 100-200$ one) is great and responsive, with an easy way to escalate if a response isn't helpful.
Hell I got a sit down with the engineers in Amazon Go (the walk out of the shops and pay automatically thing) because I was curious. As a consulting partner!
Now bigger companies have processes, hierarchies, etc. to manage risk in a distributed way, because well they are bigger. But guess what these companies also have? profits! revenue! multiple products/services that produce that revenue that can be then assigned down at the appropriate level of hierarchy that took/mitigated/managed the original risk.
Google doesn't. It's a single-product company with a single Profit Line and thousands of Loss lines and it's pretending to be something else. That's why they are so busy managing risk all the time; most of them don't have revenues to manage, aim for, use to be rewarded for.
Google is a monopoly, a rentier on the internet. They need to be broken up and repurposed to multiple, actually value-creating companies. If not for the health of the internet, at least for the mental health of its employees it seems.
(I would argue that Google's "free" products, like Android, are the worst thing that has happened to the internet; they cannot be broken up soon enough).
https://www.statista.com/statistics/289659/youtube-share-of-...
Search is 58% of revenue but I'd still call it at least a 4 product company with the "other" products contributing a decent 11%.
If you want, you can stretch their products to ads, cloud, and play store. But ads are still 80% of the whole pie. More if one accepts that some percentage of youtube subscribers are only paying to get rid of ads.
Yes! I agree with this entirely. Google is the behemoth of mediocrity, the McDonald’s of tech companies. All their products are good enough to deter competition and to keep feeding search, and not an ounce better. Humans and human attention are just grist for their mill.
I must say that's one of the most unorthodox description of what a company does I've seen.
How is a bread factory or a plumber "turning risk into money"? It sounds like you're describing an insurance company or a bank, not, you know a company like Google who produces free software & services to billions of people in exchange for monetizing their attention. Focusing on risk mitigation in a company is explicitly optimizing for protecting existing income stream at the expense of innovation, customers, and new income streams. It's not what "any company does": it's what uninspired companies who have lost their ways focus on.
It sounds like what a consultant used to "financialize" everything would say to solve Google's problems and slowly turn it into an intangible conglomerate like GE who is just the shadow of its previous self. But maybe this time has arrived for Google?
You can think about arbitrarily risk-neutral enterprises that are profitable. For example, what risk is a residential REIT taking, especially one whose properties are all insured? By contrast, you can look at a business like a casino (or a catastrophe reinsurer), where it is legitimately the case that operating income is predominantly the result of risk-taking. But when you put the two side by side, there is not a huge difference in ROIC (Return on Invested Capital), which you might've expected if profits were fundamentally a risk premium.
And while lots of businesses do hedge their inputs to some degree, it is not the case that everyone does it to the maximal possible extent.
... no? Just because you've eliminated price risk doesn't mean you've eliminated holistic market risk. For example, suppose you're an airline: if fuel prices go up, and your competitors are not hedged, it's no big deal if you're not hedged, because everyone just increases ticket prices. But suppose you ARE hedged, and your competitors aren't, and fuel prices go down, and your competition cuts ticket prices, and... uh, you're screwed, because 1) you have a bunch of fuel locked in at the higher price, and 2) nobody wants to buy your tickets with the old fuel surcharges. So you're forced to take a loss, and the hedge hasn't actually reduced business risk in this scenario. It tends to depend on the game-theoretic context whether hedging commodity inputs actually reduces systematic risk.
Seriously, hedging price risk in commodities is nearly costless, decisions not to hedge almost never have to do with the actual cost of the hedge. In the real world where things are much more complicated than a toy example, many airlines do some hedging, with the aim of reducing sensitivity of profits to short-run volatility in oil, rather than trying to truly eliminate exposure to fuel prices. But that does come as a "free lunch".
To the extent that it hedges them rationally, they have exactly the same expectation profit. But the bread factory that hedges its future wheat purchases has a lower real profit than the one that does not, assuming the price of wheat doesn't go up, because buying those futures contracts costs money. That's what it means to turn risk into money: if you are willing to bear the costs of things turning out poorly, you can avoid the costs of avoiding the chance of things going poorly, and increase your profit by the same amount.
If you actually believe that a residential REIT is zero-risk and a casino is high-risk but they have the same ROIC, you should buy REITs and short casinos and you can make a big profit. That illustrates why there's probably a flaw in your understanding (though not necessarily! you might be able to make a big profit!). The market should have the same logic, and if this is actually true, REITs will be bid up and casinos bid down until the ROIC on casinos is higher than on REITs, appropriately compensating investors.
FWIW, REITs face a lot of significant risks, eg. a regional downturn could lead to all the workers becoming unemployed and vacating or not paying rent, or a rise in interest rates could increase financing costs, or rent control might be enacted and prevent the property management from raising rents as much as financial models predict. And casinos are actually much less risky than assumed: most gambling games have statistical models for how much they pay out, and with millions of games played reality tends to approximate the models very closely (due to the central limit theorem), and they tend to take out insurance on extreme black swan events anyway. I don't know the ROICs on either of these industries in detail, but I suspect that if it's similar, it's because the actual risks to them are similar.
No, the return on an investment security (in this case, a share of publicly traded stock) is different from the return on a dollar actually invested in the underlying business (i.e., ROIC). This has a lot to do with the price action that you outline.
The rest of your comment deals with why these are bad toy examples, which is fine but not really all that interesting in context.
(As an aside, the EMH is obviously false and you don't need to look any further than meme stocks for evidence, but anyway)
For example, suppose you have two businesses, both of them have a future cash flow NPV of $1 billion. However, one of them was started in a garage with $1 million of angel money. The other one is a "fallen unicorn" that got $500 million of VC money before it finally IPOed.
The ROIC of the first company is obviously going to be something like 500x the second, but this is irrelevant to their market caps, which should be the same in textbook-EMH-land, ceteris paribus.
No, they don't. I'm wondering where people get these ideas.
GP's description is actually quite astute. The legal concept of a "corporation" exists solely to pool risk and limit downside; it originally came from shipping voyages in the early age of colonialism, where a significant portion of vessels simply sank with all hands, but one successful voyage would make everyone involved wealthy beyond belief. Basically every business endeavor you do involves some risk, because the future is unknowable. The plumber might fall into the sewers and drown, the bread factory might be contaminated with E-coli and get shut down by public health after getting sued by all the customers it got sick. But certain endeavors, the ones like the plumber or the bread factory, have largely been studied, systematized, and structured to maximize the chance of success. These are low risk, but they're also low reward, because low risk invites lots of competitors. Other endeavors like writing complex software systems or predicting financial markets fail all the time, and that's why succeeding at them is so lucrative.
On the other end of the spectrum, there are very high risk businesses with no innovation and stiff competition (e.g. dealing illegal drugs).
Your argument is just about investment diversity, not about the purpose of a company.
This made me laugh out loud. I'd say drug dealing is one of the most innovative businesses around. At the high level, cartels are constantly inventing new methods of smuggling and money laundering. At a low level, dealers were first adopters for text messengers, crypto messengers, cryptocurrency marketplaces etc. I'd say this makes them more innovative than most startups (who often just re-invent traditional business but done online). Dealers even make profits!
“Using a burner phone” is a 20+ year old trope and it was pay phones before that. Not exactly innovation.
The people that made crypto currency market places were pretty innovative. The dealers that used them weren’t. They are the equivalent of people who discovered eBay in the 90s. Early adopters, not innovators.
Plumbers make excellent money, not a "little bit". I don't know about bread factories, but I assume they do well too.
Plumbers and bread factories make good money compared to salaried W-2 employees (even compared to Googlers, sometimes), but this again is a risk/money tradeoff. When you choose to work for a big company, you are trading off the risk that your efforts will be useless vs. a guaranteed payday that lets the company keep all the rewards if they're successful. In software it's trivially easy to make the opposite risk vs. reward tradeoff: just incorporate the company yourself and get acquired, in which case you get nothing if your efforts are unsuccessful but keep the full market value of your product if they are.
> They need to be broken up and repurposed to multiple, actually value-creating companies.
These obviously can't both be true. You need to take your pick of what you critizice.
The real cost of delivering these services is written off by the company as a necessary expense. The value obtained from these services via goodwill or some other metric is currently deemed to be greater than their cost.
Is it more important to have perfectly tracked projects or projects which
ship 20% faster?
To use a programming analogy, it's like the difference between latency and jitter for real-time systems. Many real-time systems will happily sacrifice considerable amounts of average latency, in order to minimize jitter. It's far better to have a process that completes in 200 ms every single time than it is to have a process that completes in 2ms most of the time, but, occasionally takes 2000ms.
Similarly, from a managerial perspective, it's far better to have a team that gives you good visibility, allowing you to plan for a completion date (even if that date is farther into the future than you'd like) than it is to have a team that mostly finishes projects quickly, but occasionally bogs down and takes a year to finish a project that was initially estimated at two months.
The problem is that far too many organizations have neither. They don't have visibility, and they finish projects late. For these organizations, your dichotomy is a false one — better tracking is how they will ship faster.
There's a reason PMs are reviled to mildly tolerated by engineers: they believe the map is the end product, not the territory.
The correct answer to which is more important is "It depends."
There are many businesses where shipping 20% later leads to ceding first mover advantage and losing the game. Largely what's being discussed in the article.
More detailed and accurate project tracking and projection doesn't deliver quicker shipping. Effectively prioritizing outstanding work does.
They're similar but definitely not the same.
There are many businesses where shipping 20% later leads to ceding first
mover advantage and losing the game.
So in which business exactly does shipping 20% later lead to "losing the game"? If anything, the real risk is in shipping a half-baked product too soon.
The critical window isn't when something is first possible (e.g. when the first product appears), but when the combination of technical abilities, component price points, and market characteristics intersect to permit success.
To use a physical example, the iPod and iPhone were great products, but they were hits because they launched with the features they did, at a specific point in time, at a specific price point.
We know that potential for success infection point existed then, because they did build a working device and achieved success.
Consequently, if they had not completed those devices when they did (say, +20% time), another device could have achieved their success.
Maybe nobody on the planet existed other than Apple who could do it... but the possibility was provably there.
Or in simpler form: Netflix was far from the first streaming video service. They were the one who launched with the right features at the right time.
Nobody cares if you catch a small wave perfectly. What matters is catching the best wave of the day, in the brief window you have.
That's the reason that Apple, especially, has been so successful. It's not because they launched at some perfect time window. It's because they launched damn good products. If the iPhone had launched in 2008, or even 2009, it would have been just as successful. We just would have had another two years of mediocre Windows Mobile/Symbian OS/Blackberry devices.
[1]: https://www.folklore.org/StoryView.py?project=Macintosh&stor...
The graphic UI and the case design were distinctive. But what really sold it was an incredibly sophisticated ad campaign that carpet bombed key media outlets with super-expensive exposure.
The problem with Symbian/Blackberry/Palm wasn't just mediocre technology, but a lack of dazzle. The products were functional but mediocre. But they were also marketed in mediocre geeky/corporate ways that just weren't that interesting to most consumers.
The original iPhone was barely more functional. But again, exterior aesthetics and a huge marketing and PR campaign forced open a consumer market for a product that was - like the original Mac - barely finished.
The problem Google has is that its products and its branding are heading rapidly towards mediocrity. It reminds me of DEC - an engineering giant in the 70s and 80s, which forgot how to read the market and innovate in the 90s, and crashed far more quickly than anyone would have predicted.
This. The ideal thing is to be second, or even third, not first. "The pioneers get all the arrows."
But as a member of a small-ish company, I remember there was this time the sales VP insisted that we MUST have this product out by a certain tradeshow. Engineering worked hard and delivered, and although the thing had warts, it gave our company an advantage: the current market leader showed up at the next tradeshow a couple months later (e.g. "20%") with a similar product and price point, but our lead in time meant that we were able to exist as a strong competitor. This single product launch and subsequent sequels lifted our company for quite a long time.
So now when the sales guys insist we might need something by a certain date, I carefully evaluate rather than dismiss out of hand - I've learned to trust their role as well.
Oh? By this rule, then Facebook must have been quickly replaced by one of the many competitors who launched later, yes?
Facebook bought Instagram in 2012 precisely because they were a direct threat. The FTC could have blocked that; or more recently, prevented tighter integration with FB+WA.
Currently they're using lobbyists to try to push TikTok (and WeChat) out of the US market.
Yahoo imploded because it had to pander to the short-term whims of the stock market; Zuckerberg references this frequently when talking about how FB (i.e. he) maintains control of its stock and autonomy.
This is it. With the advent of broad, powerful tools you can use to automate off your codebase, there's less and less manual rote work that needs tracking, and a greater and greater percentage of creative, hard to estimate, engineer-driven work. And the best way forward is to plan releases with certain features, and flexible timelines, or plan releases with fixed timelines, and flexible features.
You seem to want estimation. And you will keep doing exactly that, "wanting", because estimation isn't something that scales to big projects or multiple ones.
If we had something like a detailed blueprint before writing a line of code and the requirements were set in stone we could get a lot closer to other engineering disciplines in terms of predictability.
With the kinds of planning and estimating processes we do actually use you’re lucky to get within a factor of two of reality.
But there's also a step back -- how long does it take you to fully design, but not program, a particular piece of software? If you give a general contractor a particular set of blueprints, he can tell you roughly how long it will take to build, both the ideal number (if there are no scheduling or shipping delays) and what it probably will end up being. But if you tell an architect you want a design for a particular size and purpose and style of building, she can also tell you roughly how long it will take to draw that up.
How long does it take to design a particular piece of software? What makes a design take a day or a month to iron out, well enough that you can get a correct estimate of the amount of time it will take to build it?
I don't think we're good at that, as a profession.
We aren't. And, as an industry, we have managed to convince ourselves that's a good thing. Agile methodologies, after all, are an attempt at making such efforts unimportant.
Estimates in civil engineering are going to look just as bad as software if it was common to keep adding floors of different sizes to a building after it was complete followed by a “pivot” of turning the building into a support structure for a suspension bridge.
Software that is as rigid as tradition engineering projects in the real world can certainly be estimated, this happens in safety critical stuff all of the time. But the estimates are long, and requirements cannot change without massive delays, which is terrible for most software use cases.
Software companies don't do this much. I've worked at only one that charged for changes, and -- surprise -- such changes were very rare.
Source: this is what my Dad did for a living, and I spent a couple of years doing it in my twenties.
The requirements changes in construction projects are child’s play compared to the ridiculous stuff in software. “Oh, now that we have this data visualization app finished, please add in support for collaborative visualization exploration with synchronized views and voice chat.”
This is such a good example of prioritizing internal desires over customer needs.
> better tracking is how they will ship faster
My long experience is that heavy tracking is strongly correlated with slower shipping. Because tracking is treated as a substitute for actual domain competence.
Worked for a few exchanges and couple of investment banks and in my youth also in a big five consultancy and GS was the only shop with archaic tech. But I can now see how you could get shadow banned with comments like this.
And usually it's just them doing things like they did before which apparently isn't necessarily a good thing (sometimes is but often isn't). I cannot fault them, given the hiring expectation. But that's the problem. There's this cult that if Google does something ot must be the right way. It's filling books.
It's similar for Amazon hire in my experience.
This is the poignant question IMO.
The texture of the internet has changed drastically since the golden age of "Googling" for things. I feel like the current search vs. SEO paradigm has become a losing battle. The bad actors have adapted to Google's algorithms and now resemble the holes in Search's strategy like some over evolved contagion. The main issues I see as a lay person are:
- Bounce time and other metrics actively incentivizing content to be obfuscated and waste user's time.
- Content theft being viable and disincentivizing high quality content that can be copied easily.
- Walled garden sites that don't want to surrender their content for ad impressions and aren't easily or impossible to index.
I feel like solutions to the above problems would involve Google killing its own golden advertisement goose.
Maybe there are high influence Googlers that do come into work and think about “organizing the world’s information . . .” but a "in way that makes Google the most money" is inevitably tacked on.
"Information wants to be free" was fine for ~1990-2010, until Google et al. took advantage of it to build walled gardens they could profit from.
In retrospect, I feel like if the in-web-standards micromonetization efforts had been adopted, we'd be in a better place today, because there would have been better revenue channels than "Whoever controls ads."
And/or differentiate and regulate search as a privileged common carrier-style business class, prevented from reusing their web scraping for other products.
Maybe just make it illegal (or actually just enforce existing laws) to link to websites that violate copyright laws (from pages with monetization) and force the market to sort things out?
It would be the end of an era for the internet in many different ways, but maybe the wild west needs to just end?
And there's a clear free speech aspect. Why should I not be able to mention the URL of any other place on the net? If there's a drug dealer in my neighborhood, there's no law (nor should there be) saying I can't tell people where the house is. Why should it be different online?
a) early internet adopters were mainly young people who didn’t have a lot of purchasing power yet
b) a complete lack of trusted means for online payment
More recently things have started to trend away from this because these two issues are now solved. You can see this in social media like Discord or Telegram who have a freemium model, movie/tv streaming like netflix or disney plus, new entrants like kagi or the general proliferation of SaaS offerings which can now sustain a premium userbase where previously you would’ve opted for an ads based model (think of the n different todolist providers etc.)
A proper micropayment system could greatly reduce or eliminate the friction of payments, but we still don't really have one.
Larry Page said almost 10 years ago (!!!) that Google's mission probably needed to be updated. That's a long time to be lost in the wilderness.
https://www.theguardian.com/technology/2014/nov/03/larry-pag...
It's pretty anodyne, but by design - it's a way to push the company towards different ways of operating by creating a pretext to say "X project is part of the new mission and here's why" from a top-down perspective.
1. https://www.geekwire.com/2015/exclusive-satya-nadella-reveal...
Which is not just anodyne, it's generic to the point of meaninglessness.
There was a major search update a few years ago that emphasized "authority". If your website was considered more reputable or an established brand, you ranked higher.
This is completely against the ethos of the internet. The internet was always about new ways to find and organize information. On the internet, CNN has no greater reputation than some random blog. Yet, if you were a mainstream brand, Google would deem you to be more authoritative than some internet-only website.
You can see this most clearly in medical queries. WebM and MayoClinic top the results, even though they're filled with generic fluff. Internet-only websites and forums dedicated to a specific illness rarely get on the front page, even though they have superior information.
The topic-specific forum with thousands of pages of detailed user-generated guides and content doesn't have the social media following, domain authority, etc. that Google considers to be "authoritative". Meanwhile, the SEO-spammed trash can easily acquire backlinks and followers to game the system.
Get rid of these authority markers and go back to a more organic web if you want to save search.
I disagree this being applicable to Search. Granted, SEO spam is ranking higher, but a lot of the "internet" today is littered with low quality content. It's important to rank higher quality content.
The second problem is misinformation. It's hard to differentiate information from misinformation. Sometimes misinformation can become information with new data.
The question is, how can an algorithm determine which content is higher quality and not misinformation. "Authority" can be one proxy signal for it.
I feel in this stage of the internet, we don't need an index over everything, but just curated content. This is hard to do for Google or Microsoft, because they'd get sued hard. Look at Section 230 case in the supreme court [1].
[1] https://news.bloomberglaw.com/us-law-week/google-case-at-sup...
As someone who has first-hand knowledge of this person and the project he led, I can say (and I'm sure everyone would expect this): he does a great job of seeing all the things wrong with the org and the company; but he fails to recognize his own mistakes and shortcomings.
I would love to see one of these kinds of blog posts where the author owns up to their own mistakes.
To return to the implicit point that helicalmix appears to be making - timmg was not saying that "a lack of self-awareness makes this post invalid" (in fact, they specifically led with "This is a good article and it does a nice job of explaining some of the problems at Google") - it was acknowledging the quality of the article, and then using the form of the article as a jumping-off point to a related observation.
A conversation which remains laser-focused on one particular topic at the expense of all others is (sometimes!) helpful in narrow technical domains - but expanding to broader context is usually more interesting and more illuminating.
It’s funny that he cites the Waze blog, which similarly gives the impression of a blameless author
I am of the opposite sentiment. I think it would do wonders. It's very easy in a mega-corp to completely lose track of what the product is, who the customers are. And my intuition is that most VPs at mega-corps think about products in the abstract and focus more on politics & empire building/strategy than actually caring about the customers - and it definitely shows for companies like Google.
Just last week I tried to spawn a GPU instance on GCP, and despite being very familiar with the space (I work on cloud infra), after 20 minutes of fighting the overly complex UI and ridiculously broken flows, I still hadn't my instance up. In the next 3 minutes, I signed up for Lambda Labs, entered a CC, clicked a single button and ssh'ed into my VM.
That's what any start-up focuses on: the product. And that's why they always win over bloated mega-corps focused on internal processes and infighting, whose employees self-select for being complacent about mediocre products as long as the monopoly keeps printing fat paychecks.
Google’s products aren’t bad because the VP’s have lost the meaning of holding your customer dear. Maybe their senior managers are stuffed with former startup founders, all visionaries in their own minds, who collect big windfalls to join, play at empire building for a few years, then quit in a huff and blog about how it’s everyone else’s fault.
This assumption that every manager has complete agency over their own meeting schedule is equally flawed. I've never attended a meeting for fun, but I've attended countless meetings that were of absolutely zero value to me just because someone up the food chain from me decided I should be there.
> who get to use already stretched resources to herd untrained, resentful managers
I'd argue that managers being untrained in customer service and, especially, resentful of being forced to do it illustrates the author's point pretty well.
Large contracts are locked behind FedRAMP compliance, DoD ILx, and so on, and I happen to know that all large cloud providers listen very closely to customers in this space.
Your harsh criticism is valid under the assumption that GCP is an unsuccessful consumer product. I think it is only partially so and features are mostly driven by larger b2b and governments, where it is successful (in the sense that it delivers most features customers care about).
If you think talking to customers for an hour is pointless, then you are part of the problem...no matter who you work for.
Doesn't the culture, as described, exacerbate certain flaws in certain people while hiding flaws of a different kind in others?
At google it's easy to see how a flawed engineer might suffer from impatience, lack of tact, inability to read the room, poor social skills, not being attentive to details or to everyone's needs. It's easy to ascribe flaws to someone who makes mistakes and, instead of writing a lengthy postmortem and installing a process, goes ahead and fixes them while pleading for everyone to move on and keep shipping.
Yet at the same time, the sins of complacency, cowardice, inability to make decisions or take calculated risks, rigidity, and aversion to hard, uncomfortable work remain well-hidden. Sometimes, they are even elevated to praise! At Google, it's better to waste everyone's time or do nothing, than take a risk and ruffle some feathers (and I'm not even talking about taking externally visible risks here--a decision to build a certain piece of infra or shut down a project).
And the cycle repeats. A brilliant but "flawed" engineer realizes they can't work in that culture. They blow up and move on. Everyone says "eh, they make mistakes, can't read the room, and are difficult to work with. Poor culture fit". Well, for them, a bunch of complacent do-nothings is also extremely difficult to work with. But the do-nothings are in charge, and they are not. So they leave. Rinse and repeat.
In each case, it is easy for one to point out flaws in the other. And in some cases, "the other" is the only one that has flaws. But in most cases it is probably a combination.
Just because a company has culture issues doesn't mean that every employee is blameless or perfect. When someone doesn't work out for a job it is hard to disentangle how much of the blame belongs on each party.
In this case, I'm fairly familiar with a lot of the facts on the ground. As I said above the article makes a good critique of the real problems Google has. But it also blames some of those problems for mistakes/failures the author himself made (in my opinion).
Since you're more familiar with the facts of the case, I can contribute nothing more than to offer a different angle to look at this.
And I'll leave the author's self-improvement journey to the author.
Which in no way impacts how right or wrong his criticisms of Google are, and is unrelated to the topic he was speaking about.
Specifically, all the issues being discussed are actually just features of Google (and the rest of FAANG really) being a mature company in a mature industry.
Google cannot (it seems) launch and maintain new services. Why? Because it already has a single, ultra successful, profitable service: advertising (primarily via search, gmail, youtube etc). When you have no existing business (say you are a start up) you have an urgent need to get one. But when you have a multi billion dollar core business asset (search), you have no urgency to create another. Instead you "urgently" need to not break your existing asset.
This is the same reason GE and AT&T etc did NOT beat Google to the online space.
This is the natural result of success.
This is the healthy, efficient, economically correct position for a company like google to be in.
People have a weird expectation that Google (etc) will be a start up forever. Or an incubator for start ups. Only thats basically functionally impossible.
So the problem here is people's expectations not being realistic and refusal to accept the reality that google (with 175000 employees) is no longer a plucky 10 person startup...
Yes it takes a large crew of SREs and SWEs to keep that firehose fully primed, but nothing close to the labour force that Google employs.
When I joined Google at the end of 2011 there was something like 20,000 full time employees. At that point it was already clearly a company that had already transitioned from a disruptor of the industry to a maintainer of the status quo. e.g. they clearly bought the ad-tech company I was an employee at just to make us shut up and go away.
The conservative review and permissions structures in place there that the writer is complaining about are in place to avoid fucking up the firehose. And they're legit important. One of the most intimidating times I had working in my career was on the Ad Exchange release rotation, where I had production access for the deployment of binaries to thousands of machines in many datacentres that produced millions out transactions per second and were responsible for buckets and buckets of revenue. Please Don't Fuck Up was like an airplane banner circling around in my skull the whole time.
Any innovations that happen at Google happen because there are very smart keen people there, and Google is constantly seeking out alternate $$-firehoses. But it's never happened and I doubt it ever will. It's not a culture accustomed to being hungry enough to work aggressively to hunt $$.
Politics is eating everything there.
Ironically many capable, intelligent and driven people end up joining Google, et al. thinking it's the place for them ...not realizing it's where you go to (mentally) retire.
Huh? How do you figure? No one makes ridiculous amounts of money just writing software unless they're incredibly lucky. This sounds like typical American thinking: "if you just work hard, you'll be the next Bill Gates or Steve Jobs!" Bullshit. For every guy like them, there's thousands of people who spent countless hours and sacrificing their best years working on stuff that went nowhere; they would have been better off working at a FAANG.
Every offer I've ever received from a series A or seed stage startup has had compensation and equity potentials that -- even when imagining a successful exit via IPO or acquire -- totalled (when amortized over a 5 year period) to 2/3rds or 1/2 of what I was getting at Google. There is literally no financial reason to go to an early stage startup as an employee vs going to a FAANG. The founders and VCs don't give enough equity for it to even be close to a "lottery ticket."
As for forming my own startup means a) having an idea that VCs want to fund b) having connections to said VCs c) foregoing income while in the prototyping phase.
I have a family to feed.
I ended up leaving my Google job simply out of job satisfaction. Finances, nope. Makes no sense.
But you're right, there is a period where the founder makes little to nothing from the startup. You need to be able to weather that. And, if you're betting on the startup making you an instant millionaire, you're betting on the wrong horse. I mean, it can happen, but it's not the most likely result. The most likely result is that it will fail, and you'll have to try again with a different startup.
The difference between successful founders and unsuccessful ones is the unsuccessful ones gave up. The successful ones almost always have a string of failures before the one that succeeded.
That said, starting your own company can be much more profitable in the long run than working for someone else. And more fulfilling.
But it's certainly not a path for everyone.
This article (much like a lot of views expressed on this site) mostly just reads as someone jumping from a startup to large-business environment for the first time, and not grokking the very legitimate and unshakable reasons such an org has for being more conservative and process-heavy. Yes, that kind of thing has to be monitored and constantly worked on to ensure it doesn't grow out of control, but some level of bureaucracy is a feature, not a bug, for a large, successful organization. At Google's scale, that "some level" is quite high.
The similarities between what MS was going through then and what Google is now facing is to me the most interesting aspect of the story. Whether you're printing money via OS or search monopoly, it seems like both the direction and timing of what's going to eventually happen to your business are almost inevitable.
MS today is quite different from what it was 10 years ago when the writer left. By getting acquired by Google in 2020, it's almost as he travelled back in time, into another Big Tech company at that same stage of the enterprise lifecycle.
I think this is an excellent question.
I think Google has tried to be an incubator etc. And it's management have had a go at new products. That means hiring
I think they have also been pretty big acquirers. That also pushes the headcount up. Cynically people also say Google acquires companies to stop those companies becoming anything. When you do that, you need to keep those employees on the books too (or they will just go re-found the thing you just paid to kill). Not sure how true that is, I leave it to the reads discretion.
When both these sources of new services have failed, they have NOT fired people. Closing Stadia did not lead to firing 90% of the "Stadia Team". Ditto 101 other initiatives and take overs. It was actually a joke on "Silicon Valley" (the TV show) that no one ever get's let go by tech giants. No matter if they are useless or incompetent.
So the head count is probably pretty bloated (I have no idea how many people are actually needed to run Google).
And that brings us to actual Activist Investors and last months lay-offs.
I think investors are finally calling for efficiency not growth. I think management are finally listening. I say finally not because I think I would have done it sooner, only because I think it makes sense in hind sight.
Tell that to Apple.
I went from startup world (mostly rails and frontend frameworks of the early teens) early in my career to google3 for about half a decade to the whole suite of public cloud stuff starting about a year ago and I really miss google's internal tooling.
It's not at all that any given component is much or at all better than any given public competitor. Indeed I think the article is right that on an individual component basis there are strong competitors to everything google has internally. But taken as a whole, I found it much easier to figure out what tool to use for a given job and get it working well for my task within google than it is with all the public tools. But it's an integration and analysis paralysis / paradox of choice and sales cycle problem, rather than a technical problem.
Do I go with a fully managed service or self-host? How well does the self-hosting work with my infrastructure? What about cost comparisons? Do these three tools work together at all? Do they support my particular sso setup? It's just so much crap before even getting to the point where the technical capabilities are important. And it is either prohibitively time consuming or just actually impossible to do any sort of objective analysis of different options, so everyone just guesses and cargo cults. "This seems to have the most mindshare in the community."
I experienced variants of some of these same problems with google internal tools, but there were easy and good answers to a large enough set of the questions that it always seemed a lot easier to get an answer and move on.
1. everything works reasonably well together and share the same cross cutting primitives (IAM, VPC).
2. you have service teams and support all under the same roof, so you get deep access to them and can get them to work together on solving your problem
3. you do not get a choice in the matter, so you do not have to waste time deliberating the choice. you also get rock bottom prices guaranteed.
this all made things much simpler to work with, much simpler to debug/get support for, and removed a lot of the churn we would've had if we had to re-evaluate services every N years.
Not really. There are many competing tools internally too, and you have to choose the appropriate one. One thing that's common though is they all integrate with everything else - monitoring, building, launching, codebase, documentation, etc. This makes it much easier to just make it work.
I frequently talk to my friend about work, and I realize most of the problems they face - even at good tech companies - are something I never have to think about. I don't mean the actually difficult problems which any Software Engineer would love to tackle, but the grungy ones.
The way I see it, Google has four core cultural problems. They are all the natural consequences of having a money-printing machine called “Ads” that has kept growing relentlessly every year, hiding all other sins.
(1) no mission, (2) no urgency, (3) delusions of exceptionalism, (4) mismanagement.
For history, my first big contribution inside Google was criticized for being "not technical enough." (and that was fair because the technology was straight forward), however it had pushed through the organizational inertia of "don't change anything" across several groups to get into product. My contribution was to skillfully push things through the org without activating the organizational antibodies that resist change at all cost. But this was not a skill that was appreciated at the time by the people that mattered.
The second thing was this;
Any disagreement with the management chain is career risk, so always say yes to the VP, and the VP says yes to the senior VP, all the way up.
People who were there when I was often didn't know me personally, but did know "of" me by my calling out management on their bogosity. I was not so enamored with "working at Google" to bury my irritation with bad faith communications and actions. To the point where I got direct feedback from my management chain that they would have an easier time in "calibration" if I was not causing so much friction. :-)
I suggested they might get a lot less friction if they stopped doing the kinds of things they were being called out on. I have never been one of those "go along to get along" people in the presence of basically evil[1] people.
When people ask me if they should work for Google this is the litmus test I give them, "Can you suppress all of your feelings of justice or ethics in the face of senior leaders making unethical choices chasing promotion, market monopoly, or additional profits for the company?" If the answer is "no" then I suggest they look to work elsewhere.
[1] I know that reads hypercritically, I don't have a good neutral way of describing people who make decisions that help advantage them at the cost of disadvantaging others. Especially when that decision is unnecessary or frivolous.
What truly "ails" an entity that prints billions? Does it take an insider to figure out? Does it take a technologist? If you go back to startup school and try to map the entity into the Business Model Canvas [0] you immediately spot the obvious peculiarity: the incongruous mix of customers (advertisers versus infrastructure users). What is the value proposition, to whom? All the money and brainpower in the world don't help if you have your moral values messed up. Its a soul thing.
It seems like Google is pretty good at releasing things, just not at following though on supporting them to become a real line of business. The perception that Google can’t release seems new and odd to me as an outsider.
Hint: I asked Amazonians. They told me it generates higher revenue for them, and so "works as required". I know nobody who wants, or expects this behaviour in a search box.
Thats a very interesting spin on "more customer service oriented" or some similar view I was responding to.
> And people who get really annoyed by these extra unwanted results will just complain on tech discussion forums, while still buying their books from Amazon anyway.
Guilty as charged. But an Onyx "boox" is in my future, and I'll be able to deal with Google, Kindle, Kobo, and Calibre sourced books through Android readers on a passive display. At that point, I will be much more interested in other sources of reading material which know "Scott Turow" is a person, not a style.
The problem I see with a lot of people in these discussions is that they think "happy customers" => "more revenue". This isn't true in many cases. Many times, pissing off customers will make more money. Just look at Comcast.
The sociopaths who run big corporations figured this out long ago and that's why these companies make money. They're able to easily take advantage of peoples' naivety and assumptions that other people are like them.
To really understand how corporations work, and why they do some of the things they do, you have to think like a sociopath.
For a while! But I've now made Amazon my second source for things because the user experience has declined so much over the years. Now I buy books via my local bookstore's website. And for anything else, I'll try to buy via the manufacturer or via something found on Google Shopping before I'll buy it from Google.
I still spend money with them, but much less than I used to. And as their devotion to profit first, customers last increases, I expect to keep edging away from them.
Enough, that you can get this insanity: three forms, all annoying:
a) volume I and II of a 3 part trilogy available, not part III, except in e.g. German
b) volumes II and III of a 3 part trilogy available, not part I.
c) either of a) or b) but with the entire 3 volume set available as one book discovered *after you buy parts I and II (or II and III) uniquely
Now, I know some of this is down to IPR and economy-specific rights management, it's not all Amazons fault, but its seriously annoying.
There is a version of the fraud in Amazon goods: its people selling $1 guides to books under the title of the book concerned, or a $1 review, or a number of scams which pass through the "too much hassle to reclaim" filter.
I've also had Kindle withdraw books from sale, after I've bought them (like the '1984' incident) which is typically because of a major IPR or production issue. Sometimes, they'd rather stop selling than be love-bombed by the spelling and format issues from readers.
I helped somebody publish through KDP and the other side of the deal, doing e-books and print-on-demand is also a nightmare of a different kind. They need you to use abstruse elements of the PDF spec, to set metadata, and positional accuracy of barcode for ISBN is amazingly finickity. All of their production tooling is built around 'most likely path' which means Windows, so doing this via Latex/Calibre/PDF tools in linux is a nightmare. I wound up taking the same work to a print-on-demand specialist I could actually talk to, to resolve most of this. The people who have to live in KDP have all kinds of hack scripts to fix PDF metadata to meet exacting demands, which seem really abitrary at times.
Sorry to be difficult, but the general deterioration of the search experience from "find best thing for the customer" to "find thing that makes Amazon maximum money" is a huge part of the problem for me. I've never received a counterfeit from Amazon, but every time I try to use it I feel the loss of trustworthiness. Maybe trust isn't a big deal to you, but it sure is to me.
I always see two conflicting takes on HN. That Google abandons their products or they haven't done anything recently. Not every product can be a worldwide success, but how do you know it until you try and fail or succeed? Failing means abandoning it.
I'd like it if Google frequently creates new products marked Beta for a long time and making them core-products depending on their success. After that, I'd also like it if we stop blaming Google for "killing" products.
The problem is Google releases new products without a Beta tag, even when they aren't sure if they will succeed or not.
I think the author is just pointing out the difficulties involved in operating a large organization. There’s certainly ways in which Google can improve, but Im not sure the author has provided any valuable insight into how to fix its deficiencies.
We’ve seen this before several times. The behemoths of yesteryears are todays laggards (IBM, HP etc.). Its the cycle of life for Tech companies.
I found the article insightful in that sense.
With Azure and those kinds of moves, a lot of MS tech clearly became more about getting the tech right rather than protecting some random other MS product. At least the .NET stuff did for sure as you saw more and more open source stuff flourish, pragmatic decisions getting made, elegant APIs, etc.
Google seems to be hopeless right now but as the author states, they need their own Nadella. Sundar Pichai seems to be one hell of a dud, clearly a Ballmer-like figure.
Who uses the code then ? An senior engineer interacts with customers via the code they ship. They see the metrics and the bugs they submit
Google in particular, took in a huge influx of people from Microsoft and Oracle, and that also ended up changing the culture, but that might be unfair to them (correlation), because they were already big companies, and Google was becoming a big company.
When you add a huge number of people, they're going to spawn thousands of projects, and most of them aren't going to ship, so the probability you're on a team that ships something tends to go down with company size increase.
The monorepo IMHO is not a problem, it's a huge boon. The google infrastructure is fantastic and still has features that you won't get on GitHub. In particular, Code Review and Code Search is much better. The build/test/CI infra is IMHO better than external alternatives (e.g. jenkins/circle/etc) Moreover, when you want to make changes that cross over N different projects, you don't have to checkout N different siloed repos, or worry about different versioning, as Google's repo (mostly) guarantees a single version of everything.
There are external libraries however that are better than the Google equivalents, especially in the world of Web frontends. It would be good for Google if they allowed each team to decide what external libraries they wanted to use for smaller scale projects (e.g. use React/Vue/Next/etc instead of say, Closure Compiler compatible internal frameworks)
The tech stack stuff tho isn't the real issue with Google, as the original essay pointed out, there's no real connection to customers and no urgency to ship anything. There's also a preference for rewriting into new stuff vs maintaining old stuff.
I will forever hear “Businessmen they drink my wine, plowmen dig my earth None of them along the line, know what any of it is worth”
differently now.
btw, the Waze founder’s (similar) goodbye post is also worth reading.
Actually he used the lyrics of an extremely famous original song, which were written by Bob Dylan :)
But I was just making a silly joke.
They also seem to have weird eventual consistency type bugs that can get you into weird states you can’t get out of, I wonder - what tech stack is this? Do they not use sql for this?
I.e. 0.0001% of accounts end up in a state (across multiple services) that was never expected, and therefore code to handle it never written, and thus is unable to ever transition because there are circular dependencies
The more likely and rational approach to the situation described by the author is seeking efficiency in support of the core profit generating business. Google is ripe for huge, painful cuts that further consolidate wealth in the hands of shareholders. If that absolutely massive R&D spend isn't doing much, or doing anything efficiently, cut it.
Personally I hate this approach and would love to see Google to return to its nimble and "Don't be evil" roots, but I have yet to identify sufficient motivation for it to do so.
Love that description - and of the 4 faults the author points out, that is the one that is going to hurt them the most.
No way a company fixes having 'no mission', 'no urgency' and 'mismanagement' if a large portion of the staff things they are exceptional at everything they do.
Also, I'd like to understand if their AI lab, Deep Mind, also suffers the same problem, or does it have a different culture? Does Deep Mind have 20% time?
I'd appreciate it if anyone within the company could explain it please.
For the decade that I've been at Google it has always been the case that quite a few people around me had 20% projects. This includes myself, my teammates, people reporting to me, people reporting to someone else doing a 20% project on my team and so on.
I did, around 2007: 8 or 9 engineers around a table. Maybe 1 or 2 had a 20%.
Is it even worse than that - you get rooted out because innovation side projects marks you out as being troublesome (as it does not align with corporate plans/goals)?
> “Respect each other” is translated into “find a way to include and agree with every person’s opinion”. In an inclusive culture (good —it doesn’t withhold information and opportunity) with very distributed ownership (bad), you rapidly get to needing approval from many people before any decision can be made.
IMO, The reason most people don't pursue 20% projects is because it's hard to be the manager, product manager, and main developer on a project that you're giving 1 day a week to. It's really, really hard. Most people don't have the skillset to do that well.
So the path of least resistance to a simple life is to just devote 100% of your time to what your manager wants you to work on, where its her job to manage, product manage/wrangle product managers, etc.
[1]: https://qz.com/116196/google-engineers-insist-20-time-is-not...
My main criticism would be the consistent references to the Waze cofounders blog post which I thought was entitled whining and probably the worst entry in this genre I’ve ever read.
I would not want my electricity company to "move fast and break things", and I like that they reserve loads of their people's thinking power and productivity on boring stuff like "risk mitigation" and "preventive maintenance". Please Google, apply the same to Search, Gmail etc.
What is difficult is how to be a company that is a utility, but that at the same time "needs" to jump aboard hype trains to stay relevant/rich so they can continue to be a utility.
What about embracing solar / nuclear instead of maintaining the rent-seeking status quo and preventing adoption of renewable energy sources.
I remember way back when Google was a Good Guy. They bought Doubleclick, and most people, myself included, thought that could be an excellent thing. Doubleclick was a nasty company, and perhaps Google will make them into something good, or at least not nasty.
In fact, the influence went the other way. Purchasing Doubleclick sealed Google's fate, and their mindset came to permeate Google, turning it into what it is now.
I consider it one of the great tragedies of the industry.
This has always been my view. Google died in 2007, the thing that's still calling itself Google since then is Doubleclick.
Startup founders who cash in, join Google, and are unpleasantly surprised to find that they are now just another middle manager who founded some startup nobody's heard of or cares about.
The empathy for the customer doesn't exactly ooze from the essay ("at the expiry of my three year mandatory retention period" == "now that I am all paid up").
Part one of the fix is vague management cliches, high minded mission statements and war metaphors. I seriously doubt Google lacks these.
And the fix for endless middle manager meddling? Endless reorgs? "Winnow the layers of middle management" (read: reorgs), and mandate that customers, support staff, and managers all waste their time on a weekly heart to heart (you'll note that the author doesn't mention going through the exercise himself).
The author was a Distinguished Engineer at Google and before founding AppSheet he was a Partner Architect at Microsoft. Holds a PhD.
I found the article very well written and would trust it over any other media reports.
Relative to yesterday's comments about "wartime CEOs and peacetime CEOs", he says:
> a soft peacetime culture
Also:
> Within Google, there is a collective delusion that the company is exceptional. And as is the case in all such delusions, the deluded ones are just mortals standing on the shoulders of the truly exceptional people who went before them and created an environment of wild success. Eventually, the exceptional environment starts to fade, but the lingering delusion has abolished humility among the mere mortals who remain.
I summarized this yesterday (quoting someone else) as "born on third base and thinks he hit a triple."
"Hard times create strong men. Strong men create good times. Good times create weak men. Weak men create hard times." ― G. Michael Hopf
https://acoup.blog/2020/01/17/collections-the-fremen-mirage-...
> Second: Consequently, people from these less settled societies are better fighters and more militarily capable than their settled or wealthier neighboring societies.
> Third: That, consequently the poorer, harder people will inevitably overrun and subjugate the richer, more prosperous communities around them.
I prefer real history to science fiction, myself. The 30s were indisputably hard times, but they produced the people who won WW II and went to the moon.
This just sounds like a factoid upholding an antiquated ideal of masculinity.
The "strong men" in that quote are those with strong will. Physical strength is secondary. The weak men are those with weak will. A successful enterprise, be it a company or an entire nation, always has a few of strong willed founders, with a vision and ability to inspire others. It creates prosperity for members of that enterprise. And every time such enterprises wither and die in the hands of weak willed risk-averse mediocrity.
And it is sad to see it vanishes. Hewlett-Packard used to be such a company, their products were exceptional on many levels. They had an incredible engineering culture and almost a cult following of users.
Today they make crappy printers and what else?
This is sad but I am not sure this arc can be avoided, success brings money and scale, and those are poison in the long run.
The market has moved on now but for a while there, if it was on a crash-cart in a hospital or related location, chances were it was HP.
The company you're looking for is "Keysight" (formerly "Agilent"): it's what's left of HP's famous test & measurement division.
They didn't ship an empty box though, did they? That's what the bureaucracy is partially designed to prevent. Can you imagine the reputation loss if some rogue company element actually shipped a branded empty box?
https://www.retrojunk.com/a/tlNWXQ9PQo/star-wars-figures-a-h...
What is Google using after "Don't be evil"?
> "... problems. They are all the natural consequences of having a money-printing machine called “Ads” that has kept growing relentlessly every year, hiding all other sins."
Right. Google has only one product that generates real profits, and it has one main user entry point. So they have to be very cautious and not break what happens at that entry point. That fuels the culture. (The other products: [1]. All either lose money or are small markets relative to ads.)
For the first time in decades, Google now faces a real threat - somebody else has technology that can disrupt their position in search. Yes, ChatGPT produces some bogus results as an answer engine right now, but a year from now? Microsoft is willing to spend a few billion dollars on that.
A six month planning cycle does seem slow, but this use of "being innovative" appears to mean "stealing resources from some other project".
My current thought process leads to virtual organizations formed using the existing employees as members of a virtualized democratic business. The CEO is voted in, the employee is the citizen, the managers are voted in.
I believe it would be different than the government we have because the government we have is possibly too large to function as it did closer to founding. For example, Walmart has more employees than the USA had people when the USA was founded.
In a democracy it's not important to have happy customers, or ship product, as long as internally everyone is popular. So we spend all our time doing politics and getting nothing done.
>> It's obvious because it is everywhere, every company and government is so risk adverse as to be useless.
clearly this is not true - there are plenty of companies, and startups, that are very much not risk adverse. Large companies become risk adverse because bad outcomes affect huge numbers of people.
I was told about it by a European but otherwise have not heard of big worker coops without looking for lists.
This coop is bigger than any organized crime like Yakuza. The town where it was founded in is entirely resident owned. I can’t go anywhere in the US without there being corporate landlords.
https://www.commondreams.org/views/2012/06/25/yes-there-alte...
Perhaps democracy is a poor word as well?
My thinking is on changing the hierarchy in a way that results in positive impacts to productivity, innovation, and employee satisfaction. Perhaps in a way similar to the formation of the USA.
The top down method currently used seems to have a rather negative impact. It is still a popularity contest, the only difference I see is where that contest is focused.
Is the preference not to want to work and make things better? Is that gone? Is the new model do what you're told and find another 10% effort job or five? Possibly. Maybe people like that, maybe they don't.
Though I prefer in life to look at potential solutions than at problems. I realize that's not normal
The point of the focus on bottom line profits is because it cannot be gamed like a popularity contest.
Banks would never add money, or subtract money, from a bank account just because of someone's fame or popularity, so they serve as the neutral arbiter.
It can of course be gamed in other ways such as prioritizing quaterly profits by neglecting longer term potential.
As an example, if Google had stronger product leadership, they would have one messaging/chat strategy instead of a dozen competing chat apps. Other Google products would integrate with and enrich that one messaging solution, providing more value for users and creating a Google suite that is more useful than the sum of its parts.
The author of this article clearly prefers the excitement and constant breakneck forward momentum of startups - and that's fair enough for him! - but that doesn't imply it is the objective ideal way for a company to be.
It depends on what you're measuring. I think Google is struggling to produce and maintain quality products. They don't seem to be struggling in terms of producing revenue.
I don't see any real evidence that this is the case. I think it has more to do with Google being an enormous corporation than anything else.
Side note, I really enjoyed the interspersed poetry excerpts in this article. Refreshing from normal articles whose call outs just have you reading the same sentence twice.
Way back in time, when employment was a lifetime gig and jumping between companies was rare, IBM invested a lot of effort in an elaborate internal mobility computer system that did this sort of thing. The system could do succession planning, training needs analysis and much more.
Companies talk tht talk on this stuff nowadays but hardly anyone does it effectively because of the core problem - when talent is scarce and employees have no loyalty, what's the point of training someone up only to see them jump ship to a competitor who gets the benefit?
This simple economic fact also underlies the strange anomolies in the marketplace for talent, such as your best bet for a big pay raise being to jump to another company rather than to strive at your current gig. Over the years, job hoppers (as long as it's not taken to excess) will do much better than loyal employees who stick it out.
I don't know if things are any different for Google. Perhaps the great pay, the gifted colleagues and the opportunity to work on the very biggest systems make Google special, the sort of place that can invest heavily in internal development and internal talent mobility.
I suspect not though, making it just as hard for Google to meet up with the article's lofty recommendations as any other mega corp.
lol. wonder why?
This doesn't appear to be true. The average time in job at FAANGs isn't low because people move around, it's because they hired so fast that the average person has been there less than a year, as the article said.
The pay is high and work relaxed, so a lot of 'local maximizers' tend to seep in, the type of people who want to do just enough and avoid any sort of extra effort or experiment with anything new. The kind of people who would choose courses based on how easy it is to get a high grade and not based on how interesting or useful it would be.
For people to innovate constantly, they need to feel a sense of pride in service to customers and a sense of purpose as to why they are doing their work. For an organization to innovate they need fear that they will go extinct if they don't. As long as the firehose of revenue that is search keeps going, they will keep sucking at everything else.
> The problem and negative impact lies in the manager ranks and intensifies at the director level and higher. Hiring interviews at this level are entirely subjective and the quality of the interviewers matters. Google Cloud in particular has grown rapidly by hiring mid-level and senior talent from every enterprise company around. In most cases, the skills needed to become a director at not-so-elite enterprise company X are not aligned with the skills needed to become an effective director at Google. And when people are hired from an elite tech company Y, is it really because they were killing it there but still got lured away, or it is because they ran out of runway there and Google was a good place to land.
Once you start hiring bad middle-managers and directors, you might as well bid the company goodbye. Like a parasite, they will intertwine themselves in the various processes to make themselves indispensable.
The premise is that people are catching up to Google while Google mismanages. Microsoft is coming out strong and refreshed (and, I agree, they are seemingly very well managed), and small competitors are nipping at their heels.
The intro of this article touts the new Bing AI search engine as a major threat to Google. But...is it? Microsoft released what is essentially a proof of concept piggybacking on OpenAI's work.
So is Google really in trouble? Let's look at this through the lens of revenue [2]:
- Google is essentially a search monopoly. It's impressive that Bing has gotten anywhere, but it's too early to say if gaining 4% marketshare in a year [1] is the sign of a lasting trend. The fact that Bing and Edge are persistent defaults with constant nags in Windows 10/11 as well as being integrated into Windows search and 90%+ of people still switch their browser and search to Google tells you a whole lot about consumers' preferences.
(On this note, remember that the better product doesn't always "win" in capitalism. For example, Microsoft Teams has more active users than Slack by far [3], but that's not because it's the better app - that's because it's part of an ecosystem and a migration path for Microsoft's army of business clients)
- GCP will be a cash cow forever, just like AWS and Azure.
- Android and the Google Play store are also forever cash cows. Android and iOS are a forever duopoly with no chance of disruption.
- YouTube: another cash cow with both ad and subscription revenue with no sign of disruption.
Sure, if Google Search fails, the company is basically less than half the size. But the idea that a company like Google is going to screw up on search in the long term isn't based on any sort of evidence in my mind.
[1] https://www.statista.com/statistics/216573/worldwide-market-...
> - GCP will be a cash cow forever, just like AWS and Azure.
GCP is still losing something like a billion dollars a year.
That is unfortunately not really possible. At the scale of Google, you have to keep up with a ton of different regulatory environments - the US and EU as large blocks which lead with regulation like GDPR, but also the 150+ other countries that have their specific demands. On top of that come all the diversity issues - something completely innocent and expected in the US and EU can become a criminal offense in other countries (e.g. Saudi-Arabia and women's or LGBT rights, China/Russia and anything democracy, India and running afoul of the BJP).
Like a landlord who doesn't work because he's getting rent money every month, and don't even make sure the appartments are in a good condition, and ignore tenants calls because he knows that if they leave he'll find new tenants anyway.
Yet, Google has no problem shutting down services that millions of users use daily (greader), and worse, tens of thousands of customers use (IoT Core). It seems their reward-and-punishment system, aka culture, has some issues.
Its a pity, I recall the days when I was so excited to use Google Cloud or just watch their keynotes. Regrettably, as the years went by it got more and more diluted.
Unlike Microsoft, the obstacle for Google is not greed or hubris, its tolerance. I hesitate to elaborate further, as the specifics are likely to attract scrutiny or censure.
That is no divergence, in other words: div u = 0, no influx or outflux, a self contained, closed system.
The bane of such organisations, not mentioned in the article, is groupthink: a powerful force that pulls you into the center of mediocre sameness. Imagine if IRS solicited feedback from yoir neighbors to determine how much taxes you pay. A religious neighbor across the street knows that you don't go to church, so he writes a negative feedback to IRS. Eventually, you're either forced to behave, or the group kicks you out. That's groupthink.
I saw a talk about an upstream change to submodule initialization in Git being made by a Google engineer and she was asked about the specifics of how submodules ordering would work for the default use case. Her response boiled down to "I don't know, how we do it is we <XYZ> and we're more interested in our use case".
Really good piece of writing though, it never loses the plot despite being a longer read.
If you are wondering about the downvotes, it's because HN'ers hate when you state the truth about the world being larger than the web startups microcosm.
Google must have tens of thousands of people not directly working on ads. What exactly is so "risky" about this "subsidized" surplus working on risky bets?
> But very few Googlers come into work thinking they serve a customer or user. They usually serve some process (“I’m responsible for reviewing privacy design”) or some technology (“I keep the CI/CD system working”). They serve their manager or their VP. They serve other employees. They will even serve some general Google technical or religious beliefs (“I am a code readability expert”, “I maintain the SWE ladder description document”)
From my short time there, this is absolutely spot-on and was a driving factor behind Google being the only place I've worked where I legitimately hated my job.
> You don’t wake up everyday thinking about how you should be doing better and how your customers deserve better and how you could be working better. Instead, you believe that things you are doing already are so perfect that they are the only way to do it. Propaganda becomes important internally and externally. When new people join your company, you indoctrinate them. You insist on doing things because “that’s the way we do it at Google”. Never mind if most people quietly complain about the overall inefficiency and incompetence
When I got into Google, our team was using a unified and centralized logging system for everything. This system made sense, though it took a bit of time to grok.
By the time I groked it a few months in, I learned that this logging system was getting deprecated and that we shouldn't use it.
The alternative? It wasn't in production yet.
Deprecating something in favor of something else that hasn't even shipped was CRAZY to me. So much value wasted.
At the time, I felt like there were examples of this EVERYWHERE. Despite having the world's largest monorepo, there was SO. MUCH. DUPLICITY. Bike-shedding to bike shed the bike shed in hopes of chasing that promo to P4 or whatever (which, let's be real, everyone's chasing for the RSU grants).
> At the junior levels, Google has a challenging interview process and the general caliber of hires is good. Most of this raw talent is wasted and their skills gradually atrophy (but enjoy some free massages to help get over that!).
The Leetcode interview they foist onto candidates is completely unnecessary and is, I believe, a major driver behind the rot that they are experiencing.
When you over-index on hiring devs who can solve LC hards, you get devs who are very good...at solving LC hards.
They'll build hella interesting data structures and algorithms that have been built, like, seven times before. But building boring "soft" stuff to help drive adoption, like documentation or bugfixes? Not impactful enough. (Documentation was not weighted heavily in promo decisions when I was there, and it showed.)
I am particularly salty about this because I was down-leveled due to a single question I didn't answer well (despite nailing everything else) on a topic that I never EVER used during my time there. The guy that asked it was a nice guy, but it was definitely trivia.
> Google Cloud in particular has grown rapidly by hiring mid-level and senior talent from every enterprise company around. In most cases, the skills needed to become a director at not-so-elite enterprise company X are not aligned with the skills needed to become an effective director at Google.
If I had to guess, these might be strategic hires to build deeper roots into their prospective/current customers to help proliferate GCP. I think AWS and Azure use the same strategy.
> And finally, for all the employees, don’t spend your time on memegen. It is a wallow chamber and all that internal finger pointing doesn’t help anything.
I wasted so much time on Memegen. I agree with Praveen. It's a huge time sink and (at the time) was about as FUD-laden as /r/popular.
Even worse than a company's obnoxious gatekeeping chickens come home to roost..
Think of the last ~20 years of so many CS undergrads spending discretionary time training for "the coding interview".
To the exclusion of building/exploring things that they want to make or are interested in.
So much shared zany student projects, open source, startups, piqued research interests, and personal enrichment that didn't happen.
Sounds like any large corporation after 20 years.
I think once a company is past its garage phase (a group of engineers in a room + a pizza), think Amazon/Facebook/Microsoft, its velocity gets reduced drastically.
capability is measured on output, and if they are not producing much, then they are not really capable. Potentially capable more likely
There are plenty of trust fund babies who are intelligent and can output economic value. Doesn’t mean they do.
In Google’s case, the individuals can be capable but the org may not be. The sum is worst than the parts.
No, it isn't. That's effectiveness or impact. As an example, I'm capable of running 10 miles per day but can't be bothered.
Can someone explain why this is religious?
In part, and I know this is 'blame the victim' I think shareholders truly are dumb, if they thought it was AGI and some god given uplift to the cloud-mind: it was never going to do what they wanted, and rather than 'Google erased' I'd say google communications/marketing erased: they really didn't prepare this package for release well.
Too much infeasible community expectations on something which was incapable of doing what they wanted.
No matter what the $100b is gone. I'd say the buyers at the bottom picked up a bargain btw.
Maybe most consumers ultimately core more about the improved search capacity of Bing than that it might say weird or offensive stuff in some prompt-engineered worse cases. Plus it doesn't need to be 100% accurate, just no less accurate than the average person's SEO-laden Google search results.
Because Bing isn't a material profit generator for Microsoft right now, AI search is a lottery ticket for Microsoft while Google is playing defense for their main revenue stream.