I think this is the more interesting point around which we should have this discussion.
https://www.reddit.com/r/apple/comments/7x0eif/how_apple_pla...
So that means there are hundreds of source repos, patch review systems, and the like. When I was there I saw teams using Phabricator, GitLab, GitHub, plain e-mailed diffs, and more. In the same way EPMs and project process varies massively across teams, so what that employee experienced is by no means the norm. Certainly I didn't find this to be true at all:
> Nothing could be worked on if it wasn’t in Radar with a priority number attached and signed off by the teams’ EPM. No room for a side project or time away from your daily duties because there were always P1s to fix.
In fact, I worked on a major feature that was keynoted one year that came out purely because of my side project work. If anything it was a frustration to enter the "real world" where PMs had so much more power and influence than they did when I was at Apple (where PMs didn't really exist, and EPMs existed only to manage project timelines and drive features that HI / ID came up with).
On the other, it promotes a lot of inefficiencies (I knew of at least 8 separate instances of Phabricator being used by different teams), and makes it very difficult to centralize core tooling and operations. It also makes it very hard for engineers to have internal mobility (you typically would have to do a full interview loop as an internal transfer, which is in stark contrast Google / FB's approach).
When I left there was talk of centralizing more tooling to attempt to alleviate some of these issues, but I don't know if any progress has been made since then.
If I was an engineer at Apple who worked on iMessage, and I heard acquaintances talk about the bugginess about iMessage, I would probably start out being defensive but then I would regret that we didn't ship a better product.
I think an EPM is less likely to feel like that, as they are not so closely tied to the actual product, which is what the Reddit poster was saying.
Keep in mind also that these EPMs themselves tend to be fairly competent engineers in their own right so these aren't MBAs making decisions, they've just moved on as their career evolved. They care about quality just as much as anyone. The difficult part is making the call of cutting a feature especially if it's a keynote feature because there's just not enough time to actually ship it because that's predicting the future. Another difficult call is what bug constitutes a delay of an OS release, especially when the bug report may not have the necessary details to indicate it might be more critical than it seems.
TLDR: It's not as easy as saying it's all the EPMs fault. There's plenty of blame to go around because nobody is perfect & predicting the future (which is what scheduling is) is very hard.
It's not clear because there have been absolute shit-show releases going back to the early days of OSX. Even the fondest-remembered releases were only so after a ton of polish, and that was with releases slipping significantly (you might see 3 years pass between major updates, and the new version would still be completely unstable).
I'm sure yearly releases don't help, especially if there isn't a postgres-type culture of "we ship what's ready", but software engineering issues at Apple go back a long time.
The ways in which this has changed affect both tech-savvy and casual users:
1. Older versions of iOS (which didn't exist back then) don't receive security fixes, so the only way to stay safe is to upgrade within days of the initial release (ideally before Apple publishes its security KB article).
2. Older versions of macOS often lag behind when it comes to security fixes. There is no clear policy on whether it is safe to stay on release n-1. Apple needs to be nudged to even document macOS security issues[1]. If you are moderately paranoid about security, you have to install all updates on day one (as on iOS).
3. iOS and macOS automatically download OS upgrades and nag you to install them.
4. There is a narrative on community sites that Apple's products are only good because Apple relentlessly kills legacy code and features (I don't even agree with the latter part). Dropping support for anything older than a year is considered a badge of honor because it means there will be "progress". The more paying customers you inconvenience, the edgier you are.
5. Apple feeds into this "old software is bad" meme by publicly LOLing at Android's adoption curve every year.
6. Emoji. I wish I was kidding, but only seeing squares when other people go out of their way to express their emotions is actually annoying.
7. If you are already on an OS that is buggy, you are more likely take the plunge and update to a brand-new OS in the hope that this time, things will be better.
[1] https://twitter.com/patrickwardle/status/953416156802703360
Until you don't (see High Sierra and the "empty root password" fiasco).
This annoys the hell out of me. So much so that I feel like throwing the device every time it happens. What a waste of energy, storage, and network data limits! I've had to delete the iOS 11 download on my device manually several times already. I make sure WiFi is off whenever I'm charging for fear that it would automatically install the update too (not sure about this) and would not be able to revert back.
If anyone on the iOS team is reading this, I'm not moving out of iOS 10 unless you provide a way to turn off the WiFi radio from Control Center (not just disconnect from the network and connect automatically when I'm in a different location, which is not what I want). And also fix the slowness issues with iOS 11. I have it on an iPhone 7 and the performance is bad. I can't spend several hours erasing the device, reinstalling the OS and re-downloading all the apps (several tens of Gigabytes) once again now that you've not only removed app transfer and sync with iTunes but also removed the apps section from it. Stop assuming things about people's connectivity and data transfer quotas and start designing for real users!
Using an Apple device has never been so frustrating for me like it has been in the past two years.
So now you have a larger volume of bugs since bug count is based on the amount of code written, assuming best case that developer quality is roughly consistent across all teams. Since you have a larger volume and only so many hours in a day, trying to ship all software at once at an annual cadence means your bug count will go up regardless of how much QA you throw at the problem (since in addition to finding issues you have to fix them).
Automated testing might help because you could catch bugs before they're even submitted. Well, automated testing is still very hard on these devices when you're building the OS. For example, how do you do something as simple as build & run unit tests? If the answer involves loading code onto a mobile device that means (at Apple scale) you need a farm of devices. Moreover, your test isn't valuable if it's running against yesterday's OS. Might not even build/run because you've now acquired a dependency on an API made available in the new OS. So now you have to co-ordinate building the software using an SDK that may be unstable (build breakages happen), run it on an OS you somehow have to pre-validate works otherwise & then have some confidence that the bug found was introduced by your commit & not some combination of build issue, compiler bug (oh yeah - the compiler changes drastically too while your building your next OS), or downstream dependency bug. Finally, on top of all this you need to make sure you can recover bricked devices in an automated fashion so that the maintenance overhead of your automated testing cluster is manageable. All of these aren't impossible problems to solve. The problem is that it requires a lot of effort & physical infrastructure. Additionally, Apple has had a very ad-hoc development environment where no 2 teams are the same (it was slowly changing to unify large pieces of infrastructure when I left) which complicates the ability to roll out central infrastructure. Additionally, there are different needs - the kernel has different testing requirements/strategies from system daemons, from drivers from 1p apps from backward compatibility testing.
When I left a couple of years ago Craig had actually directed his teams to improve the automated testing infrastructure but at the end of the day the software is growing more quickly than the investment that was made in QA. There's also only so quickly you can roll out central automation infrastructure to an organization as large as Apple so it's not as simple as "do more QA" or "add more QA resources" not to mention that "QA resources" in automated testing typically requires very skilled software engineers to design & build the systems. Allowing a longer bake time is exactly the right move here as it recognizes the need to decrease the volume of bugs while infrastructure catches up.
> However, they criticized it for having stability issues and overall sluggishness https://en.wikipedia.org/wiki/IPhone_OS_2
> iPhone 3G users reported performance and battery issues after upgrading to iOS 4 https://en.wikipedia.org/wiki/IOS_4
> The iOS 5 update was the subject of criticism for iPhone 4S users, as the upgrade caused problems with battery life, failures of SIM cards, and echoes during phone calls. https://en.wikipedia.org/wiki/IOS_5
> A study by Apteligent (formerly Crittercism) found that the rate at which apps crashed in their tests was 3.56% on iOS 8, higher than the 2% found on iOS 7.1.[81] > Forbes published several articles focusing on problems in iOS 8 regarding Wi-Fi and battery,[82] Bluetooth,[83] and calendar.[84] https://en.wikipedia.org/wiki/IOS_8
A lot of iOS9 issues received media attention: https://en.wikipedia.org/wiki/IOS_9#Problems
https://en.wikipedia.org/wiki/IOS_10#Problems
Except for iOS6 & 7 which appear to have had a milder reaction, if anything I think iOS 11 may be a more stable release compared to iOS 8, 9, & 10 in that while there are more frustrating UI bugs you can hit (auto-correct, iMessage message ordering) but the overall performance & stability hasn't suffered as badly. Also iOS has knock-on impressions from OSX stability which has had similar UI embarrassments + a couple of security-related missteps.
My point is I don't think there's any single particular cause that's identifiable as the reason behind poor SW quality. If it were the hundreds if not thousands of really smart people Apple employs would have solved the problem. At this point it's all higher-level issues & more often than not tradeoffs (e.g. reducing the features you ship every year & going away from a waterfall development model)
However, the contradiction is that single giant releases help in three ways. The first is it enables you to make 1 big splash of a years worth of work to customers rather than it being dribbled out of the course of a year across different components, which is super important because it dove-tails into the release of the next flagship phone. It also makes the messaging because you can build themes for your releases that are co-ordinated across many apps/services.
The second is that it co-ordinates large-scale changes across the company (e.g. major UI redesign, some major UX improvement, etc) which would be more difficult to co-ordinate with smaller releases or if apps were split out.
The third is that leaks/previews can be better controlled as managing a bunch of different releases that contain functional differences is harder than just maintaining bug fixes (i.e. prepping UI changes for 50 different apps that are released throughout the year in preparation for the next OS that makes it possible for those apps to run).
Finally, it simplifies development in a sense because you don't have to worry about 1P backwards compatibility. 1P apps don't have to worry about supporting more than 1 OS and OS changes can confidently break 1P apps (within reason) with new APIs without worrying about apps that haven't been updated. The latter part about breaking apps doesn't matter as much if the apps are bundled with the OS unless those apps can also deliver updates via the App Store as now you have to launch vehicles that can be tricky to co-ordinate.
Yes, a lot of this is pure business reasons for why the SW is co-ordinated but that doesn't mean that it's not valid. Business practices & SW are symbiotic aspects of a tech company as neither can exist without the other. Everything is a list of tradeoffs and priorities. Apple is apparently first experimenting to see if they can keep all the business pros of their current approach by altering their program management practices which they've already done - iOS & OSX today have far more dot releases in 3 months than they used to between major versions. Reducing scope is the approach they're taking in the interim to make the major releases more stable - will be interesting to see if this is a long-term shift or just an interim one until their QA infrastructure investments catch up to be able to keep up with development. If those approaches fail & the brand becomes at risk that may outweigh the other business reasons & result in independent app & OS releases (seems unlikely IMO).
In my experience most attempts to move these sorts of teams towards real revision control and development practice lead to an extinction event in which they practice every form of hazing possible to maintain control, requiring careful management. Think extreme code-format nitpicking (combined with a lack of automated tooling or a styleguide that would let "others" get it right), followed by appeals to whichever authority maintained their team, outright hostility, and eventually ultimatum or holding the code hostage.
What you describe is a problem occurring independent of the way changes are merged. I agree that it can be a real problem.
With Gerrit or pull requests, reviewers are clearly assigned and patch states are managed in a centralized dashboard. This makes it a lot easier for external stakeholders (managers, other development staff, etc.) to gain insight into how a team's working and whether or not they're being accepting or effective. And that's why, to me, these kinds of toxic maintainers seem to suffer an extinction event when moved to a more transparent system. The maintainers are the same, but their tool - quietly rejecting patches with little accountability - is taken away from them.
That is an extreme outlier. If your organization is run by someone like Linus, it can work, maybe.
Furthermore the way patches are submitted, discussed and reviewed has little to do with the process by which they're eventually accepted or rejected by a BFDL maintainer, unlike what the GP is claimnig.
If the kernel used say Github pull requests you'd just have Linus rejecting some of those instead of refusing to accept patch serieses submitted by E-Mail.
The GP is just railing against some bad experience with the "True Gatekeepers", as if a company that was dysfunctional enough to use E-mailed patches as some unreasonable filtering mechanism wasn't able to do so via other methods.
The advantages of the E-Mail workflow is that it's fully distributed (aside from the ML hosting), bug/patch discussion can naturally flow into one another, everyone can use their favorite client that they can script etc., and you don't need an active network connection to participate (do all your patch review on the plane), all as opposed to using some opaque web UI.
Yes, the BFDL maintainers can continue to operate in whatever system they're given, but the more transparent the system is, the more likely they are to be discovered and the issue corrected.
I do agree that the E-mail workflow offers (a few) advantages, but the lack of inherent transparency in the tooling tends to be quite troublesome. It also doesn't inherently offer easy integration with automated testing systems or easy audit and compliance logging.
It's more suited for free software projects like Linux & Git where the participants don't want to be locked out of all their historical discussions just because some company does badly in the stock market, as opposed to being 100% archived in hundreds of places via an open protocol, and trivial to migrate somewhere else.
I'm just saying the problems you had with it seem to come down to a completely dysfunctional corporate culture, not patches over E-Mail.
Sure if you're working with such monumental assholes that they're just going to passive-aggressively and conveniently discard whatever you tell them over E-Mail / IM / at the water cooler that's pertinent to their work area and need to essentially CC their manager on all correspondence, and are using PRs as a way to do that.
Then sure, then an E-Mail-based workflow probably isn't for you, although you could use git-format-patch to CC both your and their manager on every patch you send to achieve similar results.
Does it scale to huge patchsets? ... well not so much. However Gerrit is not the answer to that (in fact, no tool I've found provides a good answer).
I'm a former iOS EPM (not speaking for Apple, obviously, since I don't work there anymore), and although the Reddit commenter got the atmosphere of constant crisis right, he/she is misplacing the blame and misunderstanding the power dynamic. EPMs at Apple essentially have zero power over engineers' workload. They take the list of stuff the engineering managers said they want to get done this year and say "You guys are crazy, you'll never be able to do this without 3x the hours/manpower." Then they proceed to drive the team as hard as necessary to make sure that they actually deliver what they said they were going to deliver. That's it. The idea that there is this cabal of mighty EPMs twirling their mustaches and loading developers down with work is pretty far from reality.
It's true that you shouldn't be working on anything not in Radar (the bug tracker) but this is true anywhere you'll work. Project managers however do not sign developers up for all those radars--on the contrary--we're usually trying desperately to help you get rid of scope and get the task list down to what's actually do-able!
One of the great things that IMHO sets Apple apart is how engineering-driven they are. I've never worked anywhere else where engineers had so much freedom to decide what they're working on. The fact that they always decide to work on 3x what they can actually achieve is kind of on them. But that drive to try to do so much is part of what keeps innovation strong at Apple.
It places the blame on EPMs because EPMs are just the bearer of bad news, middlemen who carry out the whims of the UI Gods and VPs. The original commenter couldn't see those above the EPMs, the ones overseeing the BRBs and other inquisitions.
If this is atypical, I'm terrified to ever leave my large engineering organization - it's a pretty great model for the work we do.
Even working somewhere, sure, you'll know what the adjacent teams in your org are like, and maybe you'll meet some internal transfers, but the truth is you have no idea what life is like for the thousands you'll never meet. There might be broad strokes that differentiate the company from its peers, but only the vaguest outline is going to be universally or even mostly true.
Sounds like really healthy practice
Given Apple has a pile of cash so large it could do pretty much anything it wants... and have had for several years now... I’ve begun viewing their continued unwillingness to aggressively pursue opinion B as a failure of “upper middle management” who are clearly either suppressing the needs coming up from below, or in the very least, not pushing hard enough to their bosses that their teams are under resourced for achieving the companies goals.
It's not really context. The real problem here is the premise that "Apple has software quality issues" is taken as a given, without support.
Sinofsky makes the point that Apple is operating a such a huge, unfathomable scale and depth -- and this is coming from one of Microsoft's top guys in its heyday -- that even rare bugs that affect 0.01% of users translate into a "stadium full" of people, affecting their lives deeply. So, combine that with a sensationalist, anecdote-driven news cycle and you get an optics problem. He observes that "in any absolute sense," Apple's SW+HW quality has "exceeded everyone else."[1]
[1] https://twitter.com/stevesi/status/963142502604779520 (via Gruber)
I've found that quote to apply to everything. Familiarity breads contempt, even just from boredom, or wanting to shift blame for the monotony of the task you're using the tool for.
For example I think Excel is a great piece of software, but I know someone who spends all day every day in it doing dull but important spreadsheets. They're what I consider an Excel expert. But they hate Excel because every little minor issue is magnified for them because they spend all day every day in it.
Some software examples: Internet Explorer has not been winning the browser wars for a decade. I've deleted Skype after its interface had been sufficiently ruined, and I don't know anyone who still uses it for business. When iTunes got too annoying, I left its ecosystem for Spotify. It's not looking good for Siri in the battle for the smart home. Speaking of programming languages, I complain a lot about Swift, don't use it, and am moving in the general direction of IntelliJ IDEs.
Excel and C++ got lucky, they will be hard to replace.
This stuff is symptomatic of Apple's love of form over function, also manifested in dumb and costly UI gimmicks like 3D touch.
Properly designed animations are function: they give a sense of place and help in creating a navigation hierarchy instead of having things pop into and out of existence.
> iOS is just littered with gratuitous animations that needlessly slow down the interface
Their gratuitousness is evaluated by you by the "slowdown" they bring. The only slowdown they bring me is when they're not interruptible/concurrent, but that's a bug, like the calculator app's one.
> My two least favorite are the home screen zoom-in effect you get every time you unlock the device
Unless I'm literally trying to race the beam, by the time I move my finger from the home button to a position over the screen the animation has completed. The animation is interruptible and you can tap or swipe right through it.
> and the animated text on typing autocomplete suggestions
How is that slowing you down? You just type <space> which triggers the animation yet merrily continue as the animation proceeds while you are already inputting the next words.
> costly UI gimmicks like 3D touch
To each his own: I use it everyday to peek at things and multitask, it's a real timesaver.
My brain can't identify the icon I want to tap while the animation is in progress. I have to wait for it to finish then parse the screen for my target.
You just type <space> which triggers the animation yet merrily continue as the animation proceeds while you are already inputting the next words.
Again I can't identify the word I want to tap while it's moving around. I have to wait for the animation to complete, then figure out if I want to tap any of the suggestions or keep typing.
I agree animation can be functional, but both of these animations are purely there for aesthetic reasons, get old very quickly, and actively slow down my use of the device. If I have to use an iPhone for any extended period of time I turn off animations at the system level and use GBoard for typing.
As for 3D touch. I always forget it even exists because it has zero affordance. It increases the manufacturing cost of the device for very little user benefit for the typical user. I think it also forced Apple's hand on FaceID and drove the cost of their flagship device out of the range of a lot of people that would otherwise buy one.
Interesting. I sure don't parse each icon precisely as it moves but the colors and layout allow me to intuitively have my bearings and know which page I'm on and then it's muscle memory. I can see it being an issue though.
> I have to wait for the animation to complete, then figure out if I want to tap any of the suggestions or keep typing.
Oh my bad, I thought you were talking about the autocorrection bubble that has the word come down on <space> but this is really about the predictive words above the keyboard. I turned them off entirely because I basically never used them and it seems to make the keyboard itself terribly slow after some time as it computes suggestions.
> As for 3D touch I always forget it even exists because it has zero affordance.
So are keyboard shortcuts, 3D Touch is not a requirement for using the phone but it allows one to be more efficient without cluttering the interface. Also, you typically don't forget that you can scroll or pinch to zoom on an image or map or whatever, it's kind of the same deal.
Oh one more I just can't live without now: force pressing on the keyboard turns it into a touchpad for the caret, and re-forcing it starts a selection.
> It increases the manufacturing cost of the device for very little user benefit for the typical user.
When I saw it demo'd I thought "what a gimmick", yet now it gives me so much value that any device without it feels gimped to me.
> I think it also forced Apple's hand on FaceID
Woud you care to elaborate?
When browsing a menu bar or looking at a dialogue, labels will usually have an underlined letter (sometimes revealed by pressing Alt) that indicates you can press Alt+<letter> to trigger that action. Items from the menu bar will typically list their corresponding hotkeys directly.
This is true across pretty much every desktop UI toolkit, even, surprisingly, in the Electron based Slack client I have open right now.
90% of the time, I forget it's there.
A specific case of where it was a problem for me is that tapping the lockscreen icon for the flashlight does nothing. I thought it was broken for a week till I accidentally discovered that you had to force push it.
At least macOS gives you the option to disable gratuitous animations with their accessibility settings. I'm hoping eventually Android will wise up and follows their lead.
Settings -> Developer Options -> Drawing
There are various settings there for animation styles and speeds, including 0 speed ( none ).
Not really, you turn on "developer options". There is no developer mode as such, just many separate options, all with defaults that don't change things from non-developer behavior.
I have animations set to 2x regular speed, it's nice.
It's cool to see other Apple engineers and PMs chime in with their perspective.
Second, even if it's just anecdotal, I have personally been experiencing issues with my tMBP at work that I haven't had with any previous laptops going back to a G4 iBook. Stuff like random crashes, parts of the screen failing to render properly, etc. IT says that they're experiencing more of that sort of complaint with the current gen than they have with others in the past. The "sensationalist anecdote-driven news cycle" is blowing things out of proportion, to be sure, but there does seem to be some measurable degradation in their Apple's standards as of lately.
> behaviour from the PM team. I've worked at companies where that sort of behaviour was rampant, I've worked
> at places where it wasn't, and the difference in the end product is palpable.
Yeah, I'm not a fan of dismissing complains by saying that its normal for engineering to think that way about PM. There are different mindsets involved and it is definitely normal for them not to see eye to eye all of the time. It is also normal for them to want to prioritize things differently and there are some delicate balancing acts to be played between meeting legitimate engineering needs and pragmatically driving a product forward.
But sometimes, the PM really is causing problems by preventing those things from happening or by getting in the way of his own objectives.
You just characterized anecdotal evidence as "measurable". That's exactly the logical fallacy at work here.
Non-anecdotal evidence of Apple's quality standards: The super long iPad upgrade cycle. It's stretched out and impacting Apple's bottom line because >3 year old iPads still work perfect.
Is this supposed to sound bad? Because it just sounds... professional.
An issue tracker is a communication tool for use in the context of a largely self-directed activity. The only role at my company where your day-to-day activity is dictated to you by the tickets on your screen is customer support.
On my team, if you're only working the asks handed down from above and not proactively taking initiative to make things better, you're barely pulling your weight.
Taking the bigger picture into account when designing a solution is simply good engineering practice. I don't think that's what the GP was talking about.
The EM and PM are often in the room with you to support conversations with stakeholders/dependencies, absorb the brunt of cross-team communication, make tough prioritization calls, and to monitor the progress you report via the bug tracker and in weekly sprint planning sessions. The engineer is responsible for determining what part of the project to work on next, and how to spend each hour/day/week in the service of its completion. PMs/EMs are only going to challenge your plans if they're egregious, and it would be unwise to create a sprint plan that would require 100% of your time.
Most people have interviewing and mentoring to do, several previous projects that occasionally need their attention, code reviews to give, operational incidents to attend to, etc. even before you consider the undocumented tech debt fixes, refactors, and productivity investments that aren't necessarily even mentioned to PMs.
If you spend too much time on that, of course, you'll fail to complete your projects, stop getting interesting assignments, and have no hope of promotion. If you spend too little, however, your projects will turn out to be operational/maintenance nightmares that may even be worse for your reputation and prospects than vaporware.
Each engineer is their own architect and project manager, with the scale of the project tailored to the engineer's level. (You advance by earning and then succeeding at increasingly complex project assignments). Product managers help determine the vision of what the product ought to be; engineers get it there.
> Engineering managers assign projects to engineers
So engineers do only work on tasks handed down to them. We just did not agree on what "task" means.
The problem with a lot of engineers is that "make things better" often doesn't translate into "keeps customers happy and money coming in".
I could work on making our internal software "better" for the next two years but it wouldn't bring in enough money from customers to justify paying me to do it. And I'm smart enough to know that I have zero idea about what will make things "better" for the actual customers.
I can claim the same, and my own experience has been far from that poster’s experience. Any time I have volunteered to spend extra time to accomplish a goal, I have been told by my manager that it isn’t necessary for me to do so. We also do our own planning on our team and the management chain is there to help us be successful.
"EMs at Apple are powerless to push back. Every engineer's performance is tied to the number of Radars fixed and closed. Every EM's performance is tied to their team's total Radars fixed and closed, so they have an incentive to keep everyone focused on the prize."
The people who feel they are being ignored or feel they aren't being used to their full potential are going to look for someone or something to blame.
I'm sure there is some amount of "higher number of fixes is better" going on (any larger company has it), but i've seen poor team members read into things like this way too much in order to justify why they aren't doing well in comparison to others.
It may be true that apple weighs stats like that too heavily, but we aren't going to get a good picture of how heavily they weigh them from a few anecdotes.
https://www.quora.com/What-is-the-role-of-an-engineering-pro...