“A Cold War Every Day” inside Apple's internal tools group
buzzfeednews.com
buzzfeednews.com
I tell people you only call it politics when you are losing. More accurately, it's a layer of literal stupidity above the competent to shield the money side of the company from the leverage that operations people would have if they had any information about how the money side worked.
Instead of a hierarchy, rethink a company as a hub and spoke model with concentric rings. The main differences are the implication in a hierarchy that there is "gravity," keeping people down and that they need energy and leverage to climb "up," which further implies there is a place to "fall," and that there is only one way "up," instead of many possible paths to the centre from all directions. There is no gravity, only gates and barriers, and even these are just information. Politics is how a middle manager runs interference and creates distractions to make sure you can't see over, around, or through them, and that the people behind them closer to the money can't see you. Tech is usually outside the main perimeter, mediated by contracting companies or middle managers whose job is to compartmentalize the value people create, and be sure it is replaceable.
Viewed this way, of course this demented political farce is how Apple works, because it's how everything seems to work when you have internalized the precise and specific mental model someone uses to take advantage of you.
Sorry if you can't unsee it now, but hopefully it will be funny and we can get good, competent people who value tangible skills into positions of power.
Is there a community where we could discuss something like this? "The spoke." I want to get obsessed with this.
I think a simple thing to do is to spot them and favorite them if you're interested in this sort of thing.
This is a brilliant description of it. The money people aren't stupid, despite what the workers think. They just have more information and different motivations which, in most larger organisations, they actively avoid explaining to the operations people.
Edit: Changed 'an incredible description' to 'a brilliant description' because it is entirely credible.
I don't have a good model of how these things work. I encountered it first hand, but couldn't develop a good model after a long time of thinking about it. Tons of things happened that I didn't understand at all, and I couldn't understand why higher-level people wouldn't just resolve it.
Help understand this better?
If the operations people ever fully understood how badly the money people are fucking things up, they'd have leverage to use against the players on the money side and it'd be game over.
Which is why stock compensation was expected to help this, at least in theory.
This is a fancy way of saying corruption. Twisting the system to be used for your own ends, be they strategic, or just straight-up cash-in-my-pocket, is corruption.
Operations workers can see the daily inefficiencies and profit from them, or inject themselves into processes and decisions in ways that will benefit them. Adding a layer of moron/process/politics slows or halts this bad behavior, at the cost of increased overhead and inefficiency.
The problem with this is that government is easily swayed by megacorp money. That money actually creates additional regulation, first by spooking people into thinking the megacorps need to be reined in then by using their influence to ensure the congresscritters hungering for megacorp attention create regulations that are more damaging to industry upstarts than to megacorps that currently dominate their industries. As a result, all these business regulations end up strengthening the hold of industry dominators over their industries, and they just find ways to keep their games going, serving goals wildly different from what most people imagine they really serve. As a result of those regulations being so crushingly bad for small industry upstarts and basically just a drop in the bucket for megacorps, the businesses that could have unseated the current kings -- with an influx of great talent leaving the megacorps -- end up being ground up by regulatory costs until they give up and get bought out by the megacorps, then destroyed or absorbed into efforts opposed to the companies' original high ideals.
Once in a while, someone manages to break through the competition glass ceiling, but usually it's at the cost of selling the soul of the company's original vision and becoming exactly what they originally meant to make obsolete.
- The 'operations people' / 'workers' ('the competent' in the post I was replying to) are the actual work-doers who create the company's product or provide the company's services.
- The 'money people' are the people in financial and managerial positions who deal with money coming into and out of the company, and who make the financial decisions.
Usually the latter, who know how much people are paid and how much the company makes, take considerable pains to withhold this information from the former, who have specialist skills but are often not business savvy. This is because the company's profit comes from creating as large a gap as possible between the income that the company makes and the wages and other expenses that they pay.
In a small company this information is directly withheld from the staff, but they tend to have some idea. This is part of why small companies tend to either pay better than, or have smaller margins than, large companies.
In a large company, part of the purpose of middle management is to isolate the money people from the operations people so that the operations people don't find out how much they're actually worth to the company. If you find out that you're getting paid $100 to make $10,000 for the company, you're going to be pissed, and either demand a pay raise or leave.
Edit: To be clear, I don't mean to say that middle management are stupid, by any means. Part of their role is, however, ignorance (wilful or imposed, feigned or genuine) of the company's financials.
The costs of all this are enormous. Is maintaining this army really cheaper than just handing the reigns to ops people and paying them more in proportion to their increased leverage?
If you've worked in a huge org like Apple or the government, you've seen incredible inefficiencies. How can all these smart and often well-paid people be doing something so wrong (i.e. wasting time with political battles, etc.)?
But that's only true if you look at the "medium view" of the organization. If you look at the large view, how the money actually flows, then it might be beneficial for one part like IS&T to be kind of broken, as long as the rest of the company works.
I have seen this dynamic at other companies. Internal tools can sometimes hold too much leverage over the organization. They can almost "blackmail" people into getting their way because they have a literal monopoly over what they do. It might be better to let multiple teams fight battles amongst themselves, which seems inefficient if you're working as a regular joe, but could be efficient from the CEO's perspective.
----
Edit: A related idea is the principal-agent problem:
https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...
e.g. if you've worked at any large org, you know that not everyone acts in the company's interest (nor should they strictly speaking). A trivial example of this is putting extravagant meals and vacations on the company's card. (Although the nontrivial instances of this go all the way up to fraud/embezzlement.)
A less trivial example is making up fake engineering work that sounds good when review time comes, but doesn't actually advance the company's business.
So some things that seems mystifying on the ground level are pretty much a defense against this inherent conflict of interest.
In an adjacent domain, a lot of HR is a defense against employees acting in their own interest. (I see a lot of veterans on here saying: "remember HR doesn't exist for you", which is presumably a hard won experience.)
To a degree that's true. However I'm reminded of the CEO of Sears, who embraced that concept entirely. He expected each team essentially to battle for funding -- with (unsurprising) result that it was a free-for-all and teams spent more time wrangling and fighting than actually getting stuff done.
A little competition motives; bellum omnium contra omnes is just a waste of resources.
Gonna remember that one.
I was part of a team inside Apple that developed internal-facing tools for the retail teams. Our team was formed because of the extremely high cost in dollars and time that IS&T wanted to charge to develop some fairly straightforward tools. My team was taken from the departments that used the tools, and we developed things that were very custom-fit to what those departments needed.
Eventually politics overcame us, and IS&T finally managed to take over our team. We all became contractors in order to support the migration to their infrastructure, and continue development of the tools. A bit later we all became fired as our project was outsourced to contractors in India, managed by IS&T PMs who had no idea what our tool was used for and had never been inside a retail store. The tools all died shortly after that, some killed by IS&T, some petered out due to lack of use now that they no longer worked correctly or were a good fit for the users.
As far as we could tell, IS&T was run as a unique company inside Apple, which did it's fair share of price gouging in order to make itself money to keep going. Its purpose never appeared to be helping Apple customers or Apple employees, it was simply to get bigger, absorbing more money and power wherever possible, with no apparent reigning in from the parent org.
Although my experience is several years old, everything in this article rings true. The contracting companies they had us working for were taking a huge cut, the quality of the code they produced was dismal, (as soon as we were no longer allowed to re-write their code major things began breaking almost immediately) and people getting transferred around constantly and having no time to understand any one project was common. (rkho's comment about their hiring process seeming like it was simply a beard for a nepotistic contractor conversion was something we definitely saw a number of times.)
All in all it was an extremely eye-opening experience. Considering how "do it the Apple way" every other department we interacted with was, being in the IS&T buildings was like landing on an alien planet.
Another good reason to have remote support in Asia is it's a different time zone. Not for Australia obviously, but that's a harmless thing for a US company to do.
South East Asia’s a big place, and outside of some of the Philippines, Malaysia, and Singapore, English is bad.
My director offered to toss some offshore resources my way to help. Bluntly told him: I think I have 3 months work ahead of me. Give me offshore resources, it'll more likely be a 9-12 months, because by the time ive described the problem in sufficient detail, I could have just done the work. Instead, id have to review a broken "solution" that doesnt work, probably makes things worse.
Offshore is fine when you need grunt work, pounding out CRUD interfaces or already having a detailed spec. But, offshore doesnt work when you're flying by the seat of your pants.
Most of the big ossues I jad durong that release werent even my own. A few I remember: an x86/x64 inline ASM bug, use of an uninitialized variable in a DB lib that led to random failures dealing with NULLs, an overflow in a 3rd party datetime lib and ABI mismatch because of wrong datatypes used in an OSS ODBC wrapper that worked in 32-bit, but failed horribly on 64-bit.
None of the offshore talent I worked with at the time could have identified, let alone fix those issues. Several of those took reading the actual generated assembler to see what is going on. I dont write much x86/64, but I can read enough to know when shits going to hell.
The 2 big agencies I delt with were Salient and Accenture. I could work with Salient, most of their asserts were Indian and spoke English. Our Accenture resources were all Chinese. It was a pain in the ass because none of our engineering resources spoke or read/wrote English. Everything had to go both ways through a translator. Something was always lost in the translation. We ended up eating the contract and backing out with Accenture.
My partner works at a startup with a very simple CRUD project. They outsourced one of the early sections of that UI. Even in such a simple case, integrating that component has been a huge time-sink and maintaining it is basically out of the question. They're working on a plan to rebuild it, as so often happens with these things.
Fire-and-forget software simply doesn't work.
Brooks’s Law in action. Mythical Man-Month should be required reading for all developers and managers.
Within 6 months the new head was gone, the legacy code hadn’t been up kept and no new products had made full releases (of a planned 4-5). I did some maintenance and rewriting/launching one of the products I had originally started on as the only dev until they could hire someone else, but it did cost them 3x what they were previously paying me.
So it lives out of the limelight.
Also expectations of internal tools are low.
Think about the sexiest jobs in tech. They are working with the newest technology, or developing the funnest resulting product.
If you have someone working on the latest machine learning technology, management has a vested interest in keeping those folks happy.
If you have someone working on computer games, the employee has a vested interest in working hard because they're doing something really cool.
But internal tools development? Expectations from the rest of the company are low, it is viewed as a necessary evil by management but not a revenue generator, and to many employees working there, it is a stepping stone.
Which is really a shame. Good internal tools can really foster a good culture. Shitty tools really take people's minds off the ball.
The worst is when some shitty off-the-shelf tool is site-licensed and shoehorned into an organization and people just suffer. I vaguely remember working somewhere where some sales tool was used as the bug-tracker or trouble-ticket system. gah.
Along with this you have the Iron Law of Bureaucracy - organizations tend (tend!, it's not absolute) to further themselves ahead of their stated goals or directives.
The right question everyone should be asking is how does the parent organization actually make money, and how to help them do that?
0. https://en.wikipedia.org/wiki/Reductio_ad_absurdum
1. https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle's_ir...
It was a bizarre interview with three of the engineers on the team on the same call asking me very specific programming language trivia and nothing about my experience nor ability to navigate complex political structures -- you know, the things that you would expect a senior fullstack engineer to have before advancing them to the next round.
Unsurprisingly, I wasn't advanced to the next round. In light of this article, I can't help but wonder if it was all a ruse to simply demand another nepotistic FTE conversion from their same consultancy.
Fully agreed, but I don't think it is like that across Apple at all. It was like that just for that specific team OP was interviewing with, and it makes sense in the context of that whole team being Wipro converts. This is definitely unusual to have a team like that in the first place, so OP just simply got unlucky.
Given a company of Apple size, no matter how pleasant and competent an average team is, there will always be a few outlier teams that are just whack and awful to work with.
I can confidently say the same about my workplace. I like my team and org a ton (in terms of competency, how the responsibilities are assigned, etc.), and I believe that an average team in the company is very competent and high quality. However, I would be lying if I said that I haven't encountered at least a couple of teams in other orgs at the company that I would never want to work on due to similar issues.
I am curious - where do you work?
Asking so that I can potentially avoid that company, as this is the most inane and borderline backwards thing I've ever seen written about senior engineering.
That being said I agree I wouldn't want to work somewhere like what was described above. It's certainly not a style of interview I would approve of or conduct myself.
If the anecdote is related to your experiences working on FB then, please, specify it, blanket statements aren't what I expect from HN comments.
In some startups, everyone above 25 is senior developers. Everyone else is junior, there is no middle role. So it is not like companies had much choice how to label people.
Imo, most of what people write about "senior engineering" is what they personally admire in this or that senior, what they aspire to be or want others to be. It is not what actually role labeled "senior developer" is.
At my employer, the best senior developers are the ones who know they can google the answer to trivia so they don't need to remember it.
I've +20 years developing professionally. I remember the stuff I use day to day, and know how to navigate the docs of the technologies I use frequently for the less used tidbits. Being a good Senior isnt about knowing the answers to everything, but knowing how to find the answers quickly. Also, how to ask the right questions.
I don't know is always an acceptable answer, but should be followed up with "I'll get back to you in an hour or two.", or something to that effect (obviously for some problems, may need to experiment/prototype).
That's how it is at most companies.
Split orgs (IT & business) result in only work of sufficient size, scope, and impact being able to cross the barrier.
Multi-year logistics management rewrite? You'll get two IT teams.
Frank in accounting needs to schedule a daily job to copy from one datastore to another? He doesn't have permissions to, and has been doing it manually for the last 5 years.
IMHO, every org along those lines would benefit from an independent IT tiger-team whose sole job it is to find business problems and apply technology to solve them.
Kind of hard to lock down everything to personal environments, when the majority of the company are developers!
Want to encourage power users? Invest in your IT department, improve developer/engineering - IT relationships.
In my current gig, the IT department has a somewhat friendly cooperative relationship with the SRE org. The SRE's build some automation for important tasks (like onboarding) which the IT team operates; the SRE team is available to help the IT team when things go wrong with such tools. Its not the main focus of either team, but the relationship has produced a very effective IT org.
You're describing external consultantcies. Some are good and do as you describe, but some are bad and work simply to find more work.
The incentives (mostly turn time) are just too misaligned to create good solutions, even with good consultants.
Consultants also aren't empowered to fundamentally push back on a client's demands, nor have existing relationship with other IT orgs they'll need buy-in from to get durable work finished.
Like someone on a team for a decade talking about why it works so well and discusses cultural and political issues and how the team overcame them.
We tended to write "engine code." Like pipelines and whatnot.
The company had a very (very) long tradition of engineering (Like, 100 years).
There were many downsides to the way they worked (over-structured, mostly), but they always gave my team and me a great deal of respect.
They didn't use contractors very often.
To this day, design quality, code quality, product quality, documentation, and focus on deliverable are the major cornerstones of my software work, but I am shocked at how little that matters to modern development shops. It's been a really disheartening experience.
I guess I was in a silo of quality-focus, all those years. They would treat very minor bugs like extinction-level emergencies (not fun).
Modern development shops operate in a hyper competitive environment with razor thin margins. Unfortunately, great documentation is often the last thing that's prioritized for such shops.
Personally, I think this is a HUGE mistake; talented technical writers are in good supply and should be hired by any development shop to improve adoption of their product. HOWEVER, as a developer, the shop will prefer that you work on features rather than documentation.
I write about that here: https://littlegreenviper.com/miscellany/leaving-a-legacy/
1) Process-focus. It's not "stop doing that, Bob", it's "let's make sure that can't happen unless we really, really want it to". And not just for technical stuff.
2) But also a willingness to kill processes if they weren't useful. Always OK to ask "who needs this meeting?" or, understanding its purpose, "can this be a few 10-second Slacks with follow-up when necessary, instead of a 15-minute team meeting every day?"
3) Project managers who were "on the team's side", at least so far as the team could tell. You come in taking the whip to people, you end up with bullshit information. You can't manage a project effectively with bad info. You want your team to tell you when shit's going sideways, or just that it might, early. They won't do that if they think they're taking on personal risk by raising issues. And they'll be stressed out and their work will be worse.
4) Semi-relatedly, estimation based on past (team) performance, on the same project. No guess-timates becoming commitments. No "so, can you have that tomorrow? Two days?" This is one part of the "agile" thing that, when done right, I've not seen anything surpass in accuracy or in keeping panic and confusion away. It does mean your estimates at the start of a project can't usefully go very far into the future and will best be expressed as large ranges until at least a few weeks in, and that you can't swap people around all the time and keep estimating accurately—but that's true anyway, even if you pretend it's not. I have seen people care a lot that their estimates often suck but not be willing to give up the "so, one day? Two days?" ambushes or accept that there will be times in a project when responsible estimation is very imprecise. These folks tend to get bad info on top of the inherent flaws in their approaches, because they have trouble with #3.
5) No "I have five bosses" crap.
That all may be more "in the weeds" and/or obvious than what you wanted, though.
This. I've found this so imporant in my time. As far as I'm concerned, if a manager can't keep their own team on side, they might as well not bother. When the manager is popular and people unite around them in times of stress, then everything will work so much better. From there camaraderie and esprit de corps naturally starts to follow, especially if the product manager personally hired many of the members of the team and starts to build a tradition.
If I were to add something to your list it would be accountability, but not blame. If somebody does something wrong, both engineering wise but also in soft skills, its important that figures of authority take the time to both explain where the person went wrong with a view to trying to understand why the person made the mistake, but also visibly put into effect processes for the future to prevent this kind of thing, either by adding it to the onboarding training or with some kind of code review or even just make sure everyone in the team knows a key piece of information that person may have missed.
I actually laughed. What kinds of applications do people think IS&T is building? Supply chain management, HR, support software (Apple forums anyone?)... I don't see how Apple sticking its core engineering resources or at all innovating within IS&T helps their business.
Steve was the one that coined the 10X rule, or popularized it. That idea must still hold within Apple's executive ranks. I guarantee Apple's top engineers never interface with IS&T and their time won't be saved by improving it. In fact I would bet that managing the deliverable of one of IS&T's projects is a sign that you're not one of the golden ones.
That said, I obviously think these contractors should be treated much better and overworking people for low pay is never okay. Maybe Apple could use some smaller dev shops that treat their people better.
This is spot on for every large organization I ever consulted for. Moreover. Even in much smaller businesses, typically starting at around 70-100 people (sooner if management has prior large business experience) you can already see this pattern starting to seep in.
This is also why the whole private vs public sector efficiency debate is farcical. They are both identical in this aspect.
That doesn't seem like an intuitive conclusion, but it's a hard circle to square. As someone who mostly builds internal tools (though in a very different context), it certainly seems like good systems are an Archimedean lever that acts as a force multiplier across the entire org. Yet, if good internal tooling was important, you think at least one company would have been able to achieve success partially on this basis.
The consumer product industry is extremely competitive, so the counter-hypothesis of entrenched sclerotic incumbents refusing to pick up $20 bills off the sidewalk doesn't really seem likely.
Ironically and anecdotally, WeWork tried to build a strong engineering org for their internal systems before it blew up.
That is on the implicit assumption that businesses are reasonable, and accurately know how to manage tradeoffs. Your counter-hypothesis seems rather likely to me, if we look at the internal behavior of a company. When negotiating, the more dots that you need to connect before reaching "and then we get more money", the weaker of a negotiating position you have. It's the same reason why sales teams get bonuses for making sales, but developers don't get bonuses for enabling sales.
For example, we are now paying for our lack of preparation and sclerotic government with the economic and lives lost to covid-19. It just took between 3-10 years to pay the piper.
Or perhaps no one has realized the power of good internal tools, and so crappy tools are the status quo and no one needs to do better.
But as soon as one company realizes the value of internal tools and it becomes a competitive advantage, other companies in that area will need to follow suit.
They do it because it gets them promoted, and because FB and Google rely on hiring every engineer ever and giving them make-work so they won't leave and start a competitor.
In particular, do you remember what gave you the impression that Facebook "overinvested" a large amount of effort into Phabricator, that I developed and open sourced Phabricator primarily to get promoted, that I built Phabricator because of scaling concerns, or that the primary value I provided to Facebook during my employment there was just in not starting a competitor?
For example the code search, testing, code review tools are amazing and better than any commercial equivalent.
Other tools like the bug database are on par with the industry.
Others like the food, directory and interviewing tools are still pretty damn good and even if they are lacking some polish, are often far more functional than the average commercial equivalent.
And of course they are hit and miss - because there are so many of them. The ones that are commonly used are well funded and have lots of people working on them.
I no longer work at Google but still consider it a role model for how companies should take tooling seriously.
Thankfully, there is nearly always an overlap period when one solution gets deprecated and the replacement is launched.
The COVID19 office shutdown has really focused attention on our internal IT tools (most of which are also offered externally). So far, they've worked pretty great, in no small part because things like BeyondCorp [1] were designed from the start for distributed workforces in zero-trust security environments.
It wasn't always like this though. Corp-Eng, as it's called at Google was for a long time seen as a dead end career-wise - disconnected from Google's marquee products. However, a number of developments, including the rise of the enterprise and cloud computing businesses, have bolstered its internal importance, and with that have brought excitement and talent to the organization. It's now an important source of product feature ideas that eventually make their way to customers.
That said, a lot of things not core to Google's problem spaces do get solved with outside vendors' software, and some of that software is great. For example, I just used a site licensed version of the Chrome ScreenCastify [2] extension to make a video demonstrating a feature I'm developing. It worked perfectly.
But we do have a dynamic that i find interesting. The company is organised into many small business units - some very small, fewer than ten people. Each is accountable for its own profit and loss. They hire their own programmers to build the actual line-of-business software they need. But there are also internal tools and systems, shared across units or used by their developers, that are built by programmers hired directly by the top level of the company - their reporting line goes straight up to the CIO and CEO, rather than to a business unit manager. So, in a way, our company does the exact opposite of deprioritizing internal tools and systems! The programmers who develop the internal tools and systems are the elite!
Most of them learned the hard way though. And some learned the hard way and still have awful tooling despite some efforts to fix them (Bungie...).
Management of the group was also strong in my opinion.
Internal tools got just as much support as product. Engineers moved between working on both, and the hiring process was the same for both.
I did an interview recently about this on the retool blog [0]
[0] https://retool.com/blog/how-to-build-great-internal-tools-je...
Netflix is indeed becoming quite the sensation for its internal policies— the general intelligence apparently prevailing there. Seeing tools as first-grade citizens seems like a very sane approach to me— empower each other and watch people create marvels.
Papermill¹ notably is one incredibly seducing tool IMHO.
1: 2019 talk by Matthew Seal, ~40 min https://www.youtube.com/watch?v=3FmBJ847_y8
Google has a tool where interviewers can provide feedback and a numerical rating for candidates. But it also has a whole data mining workflow: you can visualize an interviewer's past ratings on a histogram, compare their hire ratio against average hire ratios, etc. It's cool and interesting, and sees a lot more love than many of their user-facing products.
Facebook has Bootcamp, their ~8 week onboarding process. There's a zoo of purpose-built tools just for Bootcamp, like the bot that sends you a question whose answer is written on the whiteboard just to confirm that you attended some Bootcamp class, or the Bootcamp Mentor rating machinery.
Some of that is excessive but Apple's underinvestment sin is far worse. The SWE org learned to run lean during the dotcom bust, and never unlearned it. Radar is your lifeblood, make it best in the world! Please!
Source: worked at all three!
iTunesConnect has had an overhaul in recent years to give it a much shinier and more reliable frontend. I believe the original was closer to the original WebObjects code, and now they've extracted out the user-facing bit into something a bit more modern. There's a lot of legacy WebObjects stuff around the iTunes backend, and my guess is that it's good engineers held back by old tech, rather than just bad engineers.
Tim Cook probably couldn't care less about the internal affairs of IS&T. As long as they do their job, play by the rules, and don't impose too much overhead on the company, it's all good.
I have an offer to join an engineering team at Apple, but the chances of getting hit by layoffs is giving me pause. More details on my situation are here: https://news.ycombinator.com/item?id=22809769
I would have thought that the newest would be the most vulnerable, since they haven't had the time to absorb institutional knowledge and make internal connections.
Thanks for your insights.
And I wonder if this IS&T power grabbing and politics happen and grow within the past 10 years. i.e Happen under Tim Cook's watch.
It surely doesn't seems very Steve Jobs approach.
IS&T was completely broken under Jobs and hadn't changed much under Cook while I was there.
Yes, but I think they have a dedicated team to it now?
That is probably why Apple software is among the worst in the industry, and it's a shame. I know IS&T isn't strictly software like iTunes but it fits the pattern I've seen at Apple. But I guess if it doesn't affect their sales maybe I'm wrong and it doesn't matter and they are right.