How Apple overcame its culture of secrecy to create AirPods Pro
fastcompany.com
fastcompany.com
All kinds of projects at Apple have their own disclosure forms, and you are only given one to sign if it's deemed necessary to your work. My responsibilities on this project didn't entitle me to be disclosed on it, which led to all manner of hilariously frustrating guessing games as I tried to deliver on the requirements without actually being told what they were. Conversations regularly went like this: "I can't tell you that that approach won't satisfy the requirements, but I would think twice if I were you." Ultimately I think that I puzzled out what was needed and successfully delivered it, but the method was pure madness.
This wasn't even some new silver gadget launch, it was an infrastructure component to a future product launch down the road. Yet everything and anything can be given the top secret treatment.
Why is that? Running a disclosure-required project is prestigious. Being disclosed on projects is a badge of honor, almost even a high score board, and not being disclosed is used as a weapon in big or small ways.
Anything that Apple manages to ship (in my experience) is in spite of their corporate culture, not because of it.
The hardware business is...
Let's just say that the difficulties you face definitely exist for hardware engineers.
But, difficulties involving negotiations plus recalcitrant South Korean factory owners are WORSE.
In "South Korean factory owner negotiations," secrecy is paramount.
Particularly if said South Korean factory owner is the biggest supplier of iPhone screens. And also makes an entire line of competitor phones to your products.
Apple's CEO, COO and "head of services" all have degrees from Duke University and 30 year careers in supply chain management. The company is widely regarded as being run by a Gang of 4, those 3 plus the general counsel.
From the outside, Apple is very much a hardware company that knows nothing about software.
The software is only good because Mac fans join the company and slave hard enough to make it that way.
To your example, imagine that the full-time Apple employee responsible for negotiating with that SK factory owner also doesn't know that Apple wants the factory to produce iPhone screens. Just go sign a factory that satisfies our hundreds of requirements, none of which you know, and by the way we need it in a month and everyone else knows what is required but they can't / won't tell you.
There is just no way to get the best results when you operate that way internally.
Keep in mind as well, all corporate policies follow a normal distribution.
Most of the time, I find corporations never aim for 90th percentile high performance. Perfect is the enemy of cost effective.
They want "works pretty well 80% of the time. And the 20% that's balls up, make sure it's not so bad."
In your case I suspect the reason is they think "too much secrecy" has much less downside than "too little secrecy."
Now, whether that's incompetence or malice, we can never know. But those are the gears turning in the head of the Director/VP who's classifying these projects.
Here's the rub for me, in this particular role at this particular time in this particular company: the workload was extremely heavy, the deadlines were extremely unrealistic, the threat of failure was extreme (up to and including terminating the entire org for failure to meet objectives), and yet it must be done blindfolded and with both hands tied behind our backs.
I'm sure it isn't always like that at Apple, but that was toxic and it contributed to all sorts of toxic behaviors throughout the org. It's no wonder to me that this behavior leads to burnout across the company.
At any one time a project or team gets chewed up and spat out.
Corporate one size fits all policies, and the Darwinian competition for resources, inevitably creates this.
That's just the story of corporate life in a nutshell. Shit goes wrong, you gotta get ready & job hop at the first sign of trouble.
For what it's worth, Apple is hugely siloed and also just plain huge. It's entirely possible that the culture in other orgs was completely different from what I experienced, because it was very hard to interact with anyone outside of your org or the current project scope.
How are they so successful despite this culture? In the case of the project I worked on, I saw a few reasons for success:
1. Management expressed that failure was not an option, so a few people (myself included) out of hundreds pushed ourselves beyond the limit to deliver.
2. Spending a TON of money. I had a different approach to Apple's way of controlling costs (they were very much in the "buying DRAM for iPhones" mindset), and easily shaved millions off of the project. But the inefficiencies inherent to the project's timeline and secrecy and other stakeholders meant the project came in probably 2-3x more expensive than I would have otherwise spent.
3. Leveraging existing institutional resources. Already having a global network and datacenter footprint helped immensely on the time to ship, but it also came with a ton of bureaucratic baggage.
4. Being so large that it ultimately didn't matter. While the project was essential for a key initiative to succeed, and many people (perhaps the entire org) would have been let go if it had failed, ultimately the company would have been fine if it didn't happen. They probably would have just postponed the launch by a year or two and had another org handle the project. It's very hard for a company Apple's size to have anything be an existential threat, so you get a lot of chances.
Something that's telling of my conjecture is the use of the phrase, "knock off airpods" in your comment - I imagine that came about subconsciously, and yet such a phrase seems to have a powerful effect on every other product that comes after.
e: formatting, I get markdown rules mixed up with HN's!
If you want an Android phone with decent software and hardware, you’re stuck with only being able to choose a Google produced phone.
It’s about protecting price on contracts for supplies that are locked in well before launch. When AirPods launch apple has contracts lined up already for production of many thousands of them if not millions.
> The hardware business is...
> Let's just say that the difficulties you face definitely exist for hardware engineers.
I’ve worked in hardware and I’ve worked with multiple Korean CMs and I’m still struggling to understand what you’re trying to say. I don’t agree that this makes “total sense”.
Instead of being intentionally vague, can you please just describe what you’re trying to explain without the “Let’s just say…” and other totally unnecessary secrecy? This entire thread is about how toxic and unproductive it is when people use unnecessary secrecy and vagueness, so it’s kind of ironic to read comments using unnecessary vagueness.
I was maybe close to making a point that with supply chains, there's many reasons that might justify that kind of secrecy.
Really I just want to create empathy about non-software engineering reasons for secrecy.
Not ram home a particular hypothesis about a corporation I've never worked for. I have a habit of thinking from the perspective of corporate titans.
re: price negotiations which is my jam (3-D Negotiation and Never Split the Difference are excellent bedtime reading).
Keep in mind secrecy is often not JUST about withholding information to negotiate the best price. Sometimes the information is something the other person in the negotiation would also like you to keep secret.
Thats sounds a bit presumtious, have you ever validated how accurate that perspective is? Maybe the perspective you imagine is totally different from reality.
But I honestly think we're very biased towards assuming our senior corporate leaders are foolish and misguided.
Saying "assuming a corporation made a smart decision, why did xxx make that decision?" is usually a very fruitful line of thought.
I honestly think we're very biased towards assuming our senior corporate leaders are gods and not fallible mortals.
All humans make mistakes. Very bad mistakes. Even very smart people. Depending on circumstances and level of power, some people suffer the consequences of their mistakes, and others don't.
One common mistake of leaders is to surround themselves with "yes men" who never criticize them or tell them the truth. Leaders can become very detached from reality, but their power allows them to survive and even thrive in a state of reality detachment. And these leaders always have people who will defend them no matter what and paint them as infallible geniuses because of their power, which is part of what contributes to never suffering the consequences of mistakes.
But when they committed to that mistake, they thought they were right at the time.
https://pubmed.ncbi.nlm.nih.gov/23771902/
Diagnostic errors by Doctors are probably the gold standard of being error free. My estimate is that CEO types are usually right at least 60% of the time with these big decisions as it were.
What I'm curious about is why they thought they were right when they made the decision.
I never met a person who commited to a mistake, and thought they were wrong at the time. Have you?
> My estimate is that CEO types are usually right at least 60% of the time with these big decisions as it were.
How do you arrive at this number? For a doctor there is a clear correct and wrong diagnosis. For a corporate CEO, who decided against doing a merger, how do you even know that decision was taken at all? How do you known if it was right or wrong? You can't simulate an alternative business.
Suppose you could simulate it accurately, and the revenue were up, profit margin is down, and stock price is up - is that good or bad? You can't even determine, non subjectively, which option is better
This may be important to determine liability in a malpractice lawsuit, but otherwise I'm not sure why we should care much why someone was mistaken if we're all agreed that the decision was mistaken.
The difference here is that we're not even agreed that Apple's culture of secrecy is wrong. It's a controversy. If we're talking about a diagnostic error by a doctor, we're assuming there's no controversy over whether it was actually wrong.
Two vastly different scenarios:
1) Tim Cook made a mistake but made the best decision he could given the information at the time.
2) Tim Cook didn't make a mistake.
If you're willing to agree on 1, and that Apple's culture should change, then I'm happy to grant you "made the best decision he could given the information at the time", or at least not argue it too much, because the culture is the important matter, and not the thoughts inside Tim Cook's brain.
"Assume he's smart" is vastly different than "Assume he's right".
There are many times in my career when I look at code that I'd previously written and wonder "What the heck was I thinking???" It's an interesting question psychologically, but still, I fully recognize that the code I wrote was bad. And regardless of how much time I spend on post-mortem analysis, the crucial thing is to fix the code.
> corporation made a smart decision
The bankers who traded subprime mortgages and fucked off with their gains made an excellent decision. The corporations didn't, because they are not real, they don't make decisions.
Just because something is in the interest of a senior leader, does not mean it's in the interest of the whole company.
I don't really see what this detail has to do with OP's point about hardware engineer logic/gang of four/etc
Lose tongues are everywhere, and Apple can't seem to keep anything secret in its China RnD unit.
Shenzhen is a city of 17M on the paper, but very few people are working in Hardware now. It's a very small industry. I feel I know more than half of all companies on somebody's resume. Most of 30-40 years olds in the industry were already working for 10+ years.
Btw Most of AirPods RnD was done in China, not California.
I would also add that hardware unit side in Apple is said to be very conservative, and a mirror image of their software team. At least in China, they hold a lifelong negative score system, where -30 is you are out under any conditions, and 15 minutes late counts as -3. They also use USB sticks to move files around, and work on offline computers to prevent leaks. Also, WinXP everywhere.
Why do you mention this? Duke awards thousands of degrees every year. It's a highly ranked university, but the number of people in the world with degrees from highly ranked universities is massive. It's not rare.
Not really. Colleges are social networks. Alumni favor each other. Facebook and Google were both founded at colleges.
> I don’t think there is an intent to malign Duke
I read it the opposite way, as attempting to say that having degrees from Duke somehow makes Apple leadership uniquely capable: "all have degrees from Duke University and 30 year careers in supply chain management".
> but it is noteworthy.
It's some interesting trivia. But it's not clear why it's noteworthy in the context of the discussion of secrecy at Apple.
Tim Cook got an MBA at Duke, Jeff Williams got an MBA at Duke. Tim Cook was at IBM, Jeff Williams was at IBM. Tim Cook joined Apple in operations, Jeff Williams joined Apple in operations. Do you think those are all just random chance coincidences?
The software hasn't been good for years now.
What does this even mean? This could easily just say "they have good software because the workers care about what they are making"
I was really surprised how uncommon this is outside of Google. People will hit an issue with a library, file a bug, and just sit there blocked without ever opening up the malfunctioning code and reading through it. Why turn a one player game into a multiplayer game unless you really have to? Reading code is what you do for a living!
Like anything, it's a skill that you need to develop, but it's one that will make you really productive.
I'm nervous about running in production software I can't open up and look at how it works under the hood. I often use that ability to understand my systems better.
Some people just don't care about that. They're used to paying a support agreement and opening a ticket or calling a support rep instead. I will never get it but clearly that's how some people like to, or are at least used to, operate.
I wonder how developers at companies like Apple and Microsoft are able to be productive with hidden code.
Personally I see a lot of value in documentation (that's how I learned the basics of programming and the systems I use), but since it's always out of date and wrong, I tend to undervalue it. Maybe that's a mistake.
As the OP says, tenting / disclosure is often used as a weapon to avoid having to get feedback on projects etc.
it's not toxic in and of itself. i think there are a lot of toxic teams but not every team and not every org are that way. if you can't get disclosed on something that you manage then you have some toxic issues but i wouldn't shit on the entire concept because of that, i would question why the org and company are that way as a whole and fix the root issues
You can say this about probably every large company out there and yet in the end they deliver. The saying "when a measure becomes a target, it ceases to be a good measure" applies to corporate cultures as well imho. Once the target becomes to optimize against the culture then the culture ceases to be a good culture.
In some ways I can see secrecy diluting the usual backstabbing politics of larger corporations since different groups lack as much information with which to wage such politics against each other.
When I was working for the defense industry, we would just vet people more extensively than needed to give us some margins and while things were on a need to know basis, you would never have been impaired on requirements like that, it’s just too costly.
You have be very rich to be that inefficient.
I think the last time I felt this way about an Apple Event was the launch of the iPad. Not only are they burning out engineers, they're optimizing for a value that isn't part of their secret sauce anymore. I expect innovation, but not surprise. An N'th generation anything just won't have the same zing as it did the first few times.
Suppose Apple had an engineering blog that released a roadmap about the M1 chips before their release. Then, Apple's laptops might not have such a massive lead on Intel/AMD processors today, as laptop manufacturers may have invested more in ARM processors by seeing them as viable due to Apple's work.
That's giving Intel a heck of a lot of credit. Intel is a massive ship moving along at full speed. They aren't the most agile of companies to say the least. Intel's x86 giant hammer convinces themselves that everything is an x86 nail. How else do you explain the fact that they are so woefully behind in the times? Apple and Intel worked together in CPUs before. Intel knows what Apple wants/needs, and refuses to give it to them. I seriously can't imagine that Apple and Intel never met about how the Intel chip just wasn't advancing, yet Intel plodded on anyways.
Whilst I acknowledge that the M1 macs are pretty good, Intel power pcs on everyone else's desks that work just fine for the standard Microsoft Office workload. Intel are not miles behind, they are just not ahead. In the world of Enterprise Applications x86s compatibility with legacy code is going to be important to many large organisations.
Meanwhile Apple with probably pull support for Rosetta2 in a couple of years (like last time), but Photoshop and Logic will keep working and the most users won't care.
So whilst Intel have not innovated at the speed us techies would have liked, x86 is here to stay in the non-apple world.
Not necessarily! A lot of office needs boil down to the Office 365 suite, Adobe and the SAP GUI - all of which now have native M1 variants. The only thing that's needed is someone other than Apple and Qualcomm actually putting out a desktop-worthy ARM CPU so that Microsoft can provide Windows-on-ARM for everyone (there are rumors of a secret deal between MS/QC that is bound to expire soon-ish). Once the price point gets low enough for corporate beancounters to buy a ton of ARM Windows devices for paper pushers, they will do so - and by that point, Intel is doomed. AMD is eating their lunch in the server market, and Intel doesn't have anything that can compete with ARM on power-per-watt and price.
On the extreme low end it's not only about Windows. Chromebooks are a thing. Eventually M1 chromebooks and Apple leveraging the fact that now they're a CPU designer? After they do some more upgrades to their own tech for their own purposes?
Then there are the applications not available on mac...Project, Power BI...
I know because amongst my estate is an rdp server for mac users to use some niche excel features and to use PowerBI desktop.
Had to do a double take there…
Most of the stuff is "officially leaked* to some places before the new releases anyway
When was the last time people were surprised by what they saw? I think it was on the iphone unveiling.
The performance per watt and rollout schedule of Apple Silicon?
ipod was innovative for the common man.
iphone was innovative for the common man.
ipad was innovative for the common man.
The rest? Meh.
Nowadays it's just an upgrade treadmill, consisting of "To re-pair my old watch I first need to update to the latest watch software (even though it worked fine before I unpaired it). But to update the watch software I need to update my phone software to a higher version than it supports, so now I need a new phone, which now won't work with my computer unless I update that, but it doesn't support the latest OS so now I need to buy a new computer..." For want of a nail.
And then say “meh”
I just don’t understand some people.
Apple’s contributions to modern computing, from GUI work, to WebKit, to AirPods is game changing.
Webkit is nice but not groundbreaking. Their GUI work isn't much better than the competition, although I do hope SwiftUI goes somewhere...
So I listed 3 products (should have been 4 products) that changed the world, but the meh is not for those (they were quite good!). The meh is for a return to 1990s Apple. Most of the existing technologies have largely plateaued and are being pushed ever higher in diminishing return increments (battery life being the final frontier). Without a product visionary, this is where we'll be for awhile.
As for the “apple can’t innovate”, I don’t buy it. Apple of the 90s was full of poorly conceived ideas. I see little difference in the “world changing product development velocity” of Apple then and now.
I do miss the flavor of old apple though, grey has somehow replaced beige as the “boring color every computer uses”
I love Apple, been a user since we had a MacPlus at home (then a Mac 2Vi, then a LC3, then PPC7600, then...) but let's not be overzealous.
Huh. And I guess stalkers quickly embraced the AirTag.
I agree, but the new MacBooks have had a fair bit of excitement around them. So maybe you just have to make several duds first?
Computers have plateaued for their current use case. The only innovation that will make a difference in the near future is AI.
I'm an Apple guy, I just thought it was funny. And my Airpod Pro observation isn't on how great they were at creating it, rather 'can you please help me get mine to switch back out of mono mode, after I put them in my ears one at a time and inadvertently convinced 'em that life is mono now?' (mind you, I kind of know: they freak out if asked to play 96k content and you get a mono low-res version)
And "regular people" definitely care about battery life in a laptop - it's probably the most important parameter for someone who genuinely needs a laptop over a desktop.
If anyone was excited about the new MacBook, I'm guessing it's because they're already deep into Apple as a brand.
Me, I just want a computer that works, has decent specs* and doesn't treat me like an imbecile, or refuse to ship Bash >3 because the legal team got antsy about GPLv3. (I just loooove when I'm writing shell scripts that might be used by coworkers running Macs and I need to make sure that I either limit myself to sh functionality, or Bash 3 functionality)
Which is why I've been running Linux laptops since then.
*And no, 16G is not enough, but I've had many fans try to explain to me why it was.
I mean, I use Fish, but when I'm writing shell scripts for others, it's Bash, because it's a common default. But now I'm limited to Bash 3.
But I'm glad that they got rid of that stupid bar, and let you get a laptop with 32G of RAM now. So kind of Apple!
And now I'm needing to sniff your OS, detect whether or not Homebrew is installed, install it if not, then install Bash >3, then run the actual meat of the script.
All because of Apple's lawyers being scared of GPLv3.
At this point, I may as well insist that you install pyenv and Poetry and deliver it as a Python package.
You already have to deal with platform-specific differences even (nay - especially!) across Linux with things like apt vs yum vs dnf vs pacman vs Nix. One more is really a non-issue.
OpenBSD uses pdksh, Busybox/NetBSD/Minix use ash, there are some linux distros that use zsh, OpenSolaris is ksh, etc.
If your development team doesn't have the time and resources to support a multi-platform development environment, they shouldn't do so.
Apple didn't change their business model, the GPL changed their license, and it's a license that's simply not compatible with any closed-source software business. Does your employer let you integrate GPL v3 code into your products?
Well neither would the code we're developing either so, not really a problem.
> Does your employer let you integrate GPL v3 code into your products?
Oh, most definitely. But then none of our products are closed-source, so we're an outlier, admittedly.
So, I don’t think it makes sense to criticize macOS for being different than Linux. I can’t run my systemd units in macOS, either.
We might as well raise hell about Apple not pre-installing Firefox for us.
As a rule of thumb for bidding purposes, we assumed that classifying a project doubled the cost.
There is a similar statement in the "Skunk Works" book.
This sounds off. When I worked there 7 years ago there was an internal website where you punch in the person’s name and it tells you about all the mutual disclosures you have. I would expect it’s integrated into Directory by now.
Also, I don’t recall AirPods development as being particularly more open considering I was in the same rough org and the only reason I heard about it in advance of launch was because I happened to be in an exec review where it was being demoed along with other forward looking products. I guess that could be considered “more open” but that wasn’t an experience many engineers got.
First half of the article: "Teams were innovating for months in silos only to finally converge in the eleventh hour before launch, ending up in five- or six-hour-long daily meetings, causing tremendous friction and burnout. People were frustrated."
Second half of the article: "regular cross-staff sessions, transparency, and shared voice" created the AirPods Pro
Frankly, pretty skimpy on the details – ironic considering the topic.
0: https://www.idropnews.com/news/homepod-setup-process-leaked-...
1: https://igamesnews.com/mobile/ios-15-4-leaks-possible-apple-...
Every single time I read a similar story and I had first or second hand knowledge of how things actually played out, the role of the person telling the story was massively stretched out.
It’s not only exciting to know what’s cooking in ankthwr group’s kitchen, it’s also great when experts from other teams volunteer to help when you’re stuck on a problem that’s their domain.
In larger companies, knowing what others are working on makes it a lot easier to find experts at something. And I’ve never experienced any of them being reluctant to share due to fear of disclosing information about the products they’re working on.
I don’t think I’d feel at home at Apple.
And, maybe you’re not as close with your brother as grandparent. Some siblings are very close.
Don't hold your breath for them to do anything, but a "reader mode" browser extension can help on your side.
I'm guessing most companies have a more leaky abstraction approach
https://gist.github.com/kislayverma/d48b84db1ac5d737715e8319...
I’ll be thinking about a solution to work around a limitation of an AWS service, reach out to the service team to get their thoughts and they will tell me, they are working on something similar. They will either tell me they are working on it or for me to go for it. I’ve never had an instance where I found out six months later that I wasted my time because they wouldn’t share their roadmap with me.
Those that sometimes actually do talk to the business are called "HR business partners".
As opposed to the people working on company-wide stuff like salary guidelines, rulebooks, “the way we work” brochures, recruitment, etc
It's a dumb title, but very common. I think the root cause is a lack of a general job title for a senior contributor in that field. With the debasement of the word engineer, I'm just glad they're not called senior HR engineer.
However, I suspect that a submission with the original source's headline would have led to less engagement (as it is "How Apple overcame its culture of secrecy to create AirPods Pro," as opposed to " Apple’s secrecy created engineer burnout").