The Sad State of Logging Bugs for Apple | Hacker News Reader

The Sad State of Logging Bugs for Apple

corbinstreehouse.com

263 pointsfireflies_88 comments

Disclaimer, I worked with Corbin for a decade, and have tons of respect for him. I agree with the diagnosis here, and I have one additional treatment suggestion:

The single easiest and most effective thing Apple could do to improve its SWE org is to invest in Radar.

Radar's importance within Apple cannot be overstated. It subsumes what would be multiple tools in other orgs. As an Apple SWE you spend a massive amount of time in it. And yet Apple treats Radar as a cost center, developed by an outsourced offshore team. It's slow to search, supports only plain text, is hard to script, and is missing obvious features, e.g. automatic duplicate finding.

Hire five good SWEs, give them a mandate to serve the needs of the org, and you will massively increase the effectiveness of every other engineer.

(My experience is now rather stale and I hope they've already formed such a team!)

> developed by an outsourced offshore team

I thought there were some engineers who worked on Radar, but then I heard that they had moved development offshore. Is this a new thing?

> is missing obvious features, e.g. automatic duplicate finding

Last I heard, I think someone was working on this in their free time. I don't think anyone has tried to add this as an official feature, though.

My experience is also out of date but I know they gave it a facelift recently to make it a more "modern" Mac/iOS app.

Anecdotally, I didn't find the situation internally to be much better. Many bugs internally go unanswered because there is not enough time allocated to fixing core systems and designing better replacements. The truth is, I know personally of several teams that aren't able to get through the queue of internally filed and scheduled bugs.

To me, it feels like Apple hasn't resourced core pieces of infrastructure and engineering teams in line with upper management's plans for growth. While many teams are relatively sequestered, once you start talking to folks elsewhere in the company it becomes clear that many teams are struggling to stay above water. More still, everyone shrugs about it because it's not clear exactly what is wrong. The best description I've heard is in many cases engineers are willing to offer hacks as a solution to meet management's demands, and management is either willing to accept those hacks or doesn't know better. That is far from a full picture, and its an example drawn from a small slice of an enormous company. But it seems telling to me. I find it completely reasonable to imagine that most teams in a place to deal with customer facing bugs don't have adequate time to do so. Not to their satisfaction, never mind customers.

At the same time, I think it can be hard to appreciate the ways in which Apple is ahead of the curve as far as the categories of software projects it tackles. So I don't mean to imply that anyone is really to blame per se. And it's also a shame because Radar was the greatest bug tracker I have ever used. It is unclear to me if Apple doesn't prioritize menial things like handling external bug reports, or if problems like that are not visible to / perceived by those in the company who could do something about it.

I wonder if it's as simple as this: a company that's making so much money just has no incentive to improve. After all, if the end goal of a company like Apple is to make money, they're already doing it.

Improving processes and hiring new people shouldn't be a big dent to their massive revenues though.

probably not, but from my anecdotal experience with upper managers (i don’t work at apple), every little bit counts towards “hitting the targets”...it seems unfortunately almost nowhere is immune to this pathological way of thinking...

> I wonder if it's as simple as this

The answer to questions of this form is almost always “no”

What made radar stand out for you compared to other tools?

Apple's culture is such that everything is in Radar. EVERYTHING. It doesn't get done if it's not in Radar.

Positive: Everything is searchable. It is organizational history going back decades. The company is built around this tool. Negative: The importance of such a tool goes unrecognized compared to the criticality of the system to Apple as a whole. So when it's slow or breaks, everyone is having a bad day.

I agree with you - Radar was a great bug tracking system. The fault isn't with the program, but the process. -- corbin

> …it becomes clear that many teams are struggling to stay above water. More still, everyone shrugs about it because it's not clear exactly what is wrong…

Nail on the head.

I was on the team that wrote the requirements for the original version of Radar, in 1988. At that time each department was using it's own flat file database for bugs. Also at that time, there was a very large testing group. All testers at Apple were together: more than 400 testers. We had a strong culture of testing. We had pride in ourselves and our company. This was pretty much thrown away when the group was disbanded in 1989. Professional testing culture was replaced with the attitude that only developers mattered. Hurray for "Total Quality Management" and later Agile, I imagine. I was gone by '91.

We originally designed Radar so that bugs would be verified as closed by the person with most interest in seeing this happen: the tester assigned to that part of that project. Then management swooped in with an edict that bugs must be verified as closed by whomever originally reported them. This is a stupid idea, because it creates the perverse incentive that no one should report a problem if they are outside the team (because then you are committing to verify the fix, which just means more work for you that has nothing to do with any of your main responsibilities).

When I pointed out that the system would now discourage people on different teams from helping each other, the sponsoring director said "that's what pink slips are for." Direct quote. Soon after that I resigned from the design team.

Without reasonably skilled and principled leadership, you just don't get quality software. And "quality is everyone's job" is just an empty and childish slogan. Excellence is not transmitted through slogans and wishful thinking. You have to assign responsibility, provide resources and time (which means lowering velocity of new development), and follow-up.

The fundamental reason why it doesn't happen is the technology market is not efficient. Quality is, in fact, not as important as career testers wish it were. You can get away with doing terrible work and not lose your job. The fact that Apple pays no significant penalties for having buggy products insulates it from our slings and arrows.

I disagree. The person who wrote the bug is in the best position to decide whether the problem was actually fixed.

I could easily see it go either way; it really depends on the bug. Some obvious issue, like "this button should do X but it does Y" can be verified by almost anyone. But some issues need the attention of the original author to really verify the bug. Maybe what needs to be done is someone in QA needs to attempt to verify the bug, or "pre-verify", and then it goes back to the originator for final verification, who can also verify it, or simply close it if they feel like QA did a good job.

That sounds like a good idea. As long as the originator has the option of having the last word, internal attempts to verify can only help.

I worked at Apple from ~1995-2002. The status quo described here, where the original bug reporter has to verify and close the bug was a beautiful thing. Nobody saw it as a disincentive to file bugs. It's not "extra work" to confirm that the team you begged to improve software had actually done what you suggested. The alternative, that software teams internally determine that the bug has been "fixed" without understanding nuances of the original report, is much worse. I love Apple's standard that the original bug reporter should be responsible for closing a bug, and I only wish that the privilege extended to third-party bug reports made via Bug Reporter.

When a process is annoying and you do nothing, people eventually do give up and leave. When it reaches that point, they’re not coming back even if you finally wake up and fix what bugged them.

Apple software quality is in serious danger precisely because of this type of community and infrastructure rot. They are not encouraging developers to help them, and a not-surprising number of serious issues have shown up in released products in recent years.

I am one of those people who gave up on BugReporter. Lots has been written on how tedious it can be but even its “rewritten” version a few years ago is frustrating (basically you could be reporting a damned spelling error and they would still want a “sysdiagnose”). Bugs stay open for years and still can somehow be Duplicate. Meanwhile, very obvious broken features make it into new versions.

This is extremely frustrating to me as a macOS developer.

I put a lot of effort into my app. I spend about 50% of my time on triaging / fixing / preventing bugs. We check every crash report that is logged, and we attempt to reproduce every issue reported by customers as quickly as possible.

By now, a significant fraction of bugs are bugs in Apple's frameworks. We try to report them to Apple, but they are ignored, or simply closed because they are related to deprecated APIs.

Of course, customers don't complain that Apple frameworks are buggy -- they complain that our app crashes! So Apple has no incentive to fix it. (If the bug affects Apple apps, it has a higher chance of being fixed. If a framework isn't used a lot by Apple, it will be full of bugs)

So we have dozens of workarounds in our code, and have stopped using some frameworks altogether (if possible).

But I can confirm that Corbin was quick to review bugs -- NSTableView related bugs were the only ones were I got fast feedback.

Android development is similar, unfortunately.

I received an email yesterday that a bug I filed in 2013 in the Android Google maps SDK has been assigned to someone. Might as well just not fix it now.

Additionally, something will work with the stock SDK, but will break on Samsung devices (e.g. camera SDK).

customers don't complain that Apple frameworks are buggy -- they complain that our app crashes -> if you try to mention this on apple's developer forum you will get banned. Apple doesn't like damaging their brand. As you have said customer blames us, developers.

It's buried in the article:

When I worked at Apple I practiced what I am now preaching. I would screen all my bugs within a day or so. I would verify bugs within a week or so. I would poke around at code and attempt to make theoretical fixes for bugs I couldn’t reproduce. I felt like it was part of my job, and I just did these things as a daily task.

From Corbin's about page https://www.corbinstreehouse.com/blog/about/ :

For 13 years I worked at Apple on Cocoa, mostly doing UI implementation in AppKit, such as NSTableView, NSWindow, NSVisualEffectView. I also did quite a bit of work in UIKit – mainly on the early iPhone releases, but also helping with the first few versions of the public SDK after that. I wrote and worked on the base classes, such as UILabel and UITableView. You may recall seeing me at WWDC in the labs, or up on stage giving a talk. Before Apple, I worked at Borland, primarily on Delphi.

The entire value of WWDC is going to the Labs, giving an Apple engineer your Radar number, and having them read you or paraphrase the internal-only notes attached to the ticket. Half the time the question/bug will be clearly resolved internally or a workaround posted. But no updates are added to the public ticket, and it will remain open and unchanged for years.

I gave up reporting bugs because they were ignored and not even acknowledged let alone fixed - for years. There is no evidence they haven’t gone to /dev/null. I won’t bother in future.

With Free software, I just do the work myself and include a patch. But beyond the tongue-in-cheek possibility of scripting things to submit every bug thousands of times to make it ‘popular’, it sounds like there’s no workaround to make an Apple bug report worth the time taken to submit it.

Yes, I also gave up more or less. I recently had a problem with AppKit, created a small example to reproduce, and used one of my Developer Incidents. I got an acknowledgement from them, was asked to open a radar, and provide the ticket number back. I did that but from that point on on there was silence. It may be possible that it will be fixed for 10.15, but the experience from my point of view is lousy.

Logged a bug in Xcode that was breaking our CI. This was never fixed.

It was frustratingly easy to reproduce. Clear the process is broken internally (as source article says) as well as externally. I guess when you sit on the highest volume pipe of money coming out of wallets for consumer tech, there is no reason to change?

Yeah, I used to report those to Blackberry, but when they owned the entire mobile field, noone gave a shit that you can't you get 'your code' on their platform.

Oh yeah, I can remember when trying to deal with Blackberry over a decade ago, when they were still relevant.

Ultimately, we just gave up and didn't release our app on their platform at all.

When their BB10 platform rolled out, the first thing they did was to make the code deployment process simpler.

> I guess when you sit on the highest volume pipe of money coming out of wallets for consumer tech, there is no reason to change?

This reminds me of the concept of "f* you" money that lets employees quit their jobs. It just dawned on me that some companies have that as well, w.r.t. their customers.

Isn't this how bugs are handled in most software companies? I have never seen anyone ever fix a bug that I report as a user. I have been asked for plenty of irrelevant diagnostics logs, seen bug tickets get relegated to low priority or won't fix, or closed because the ticket wasn't updated, though. (User perspective: I shouldn't have to nag. Dev perspective: Maybe it fixed itself or the user doesn't care anymore.) Plenty of that. And I'm on the other end of that at work. I'm a developer who must handle issue tickets, and there are tickets that have been sitting there for months, and might sit there forever unless my boss nags me to deal with them, because there are always more pressing issues to deal with, not to mention tasks that are far more rewarding to work on. I think there are limits to how much you can torture or bore a developer before they just up and quit.

Yeah, that's pretty much how bugs seem to be handled at most large software companies/projects, though they all do it a bit differently. Google is famous for making it next to impossible to speak to a human, instead sending obtuse automated replies. Apple does "radar to /dev/null." Linux? I wouldn't even consider wading into the cesspit of toxicity and petty dictatorship that is LKML.

On a positive note, I have had some great experiences interacting with the maintainers of smaller projects, and even some larger ones like Emacs. But massive companies and projects? Not worth the effort.

> Or, they just won’t want to install the latest OS because it doesn’t have anything they want or need (that’s me, I run 10.13).

(Context is Apple asking bug originators to verify whether or not the bug still occurs in the latest OS.)

This seems contradictory to the entire article. The point of logging a bug at all is to get it fixed. The fix would be coming in a future release. If you don’t want to take the latest updates that’s fine, but how are you going to get bug fixes?

I wonder if the author mean that he's stuck on OS X 10.13.n, and would be willing to install OS X 10.13.(n+1), just not 10.14?

Apple doesn't do point upgrades for previous releases unless there's a critical security bug in them though, so that wouldn't happen. If you want the problem fixed you have to jump to the next major release because that's what 90% of the active install base is on, since it's free and easy to install.

> Apple doesn't do point upgrades for previous releases unless there's a critical security bug in them

This is not true. Minor updates to Mac OS X pretty much always contain a handful of non-security bug fixes.

I have never seen Apple push an update to a previous version of macOS unless there was some critical bug or security vulnerability in it.

Many minor macOS updates add features. 10.6.6 added the App Store. 10.7.2 added iCloud. More recently, 10.14.3 [0] added things as small as "a menu item to News for opening a story in Safari". (As an arguably even more trivial example, 10.14.1 added new emoji.)

Most minor updates also fix small, non-critical bugs, like 10.13.4 [1], which "fixes an issue that may prevent web link previews from appearing in Messages".

Do all of these updates fix security bugs too? Yes, because a company as big as Apple is probably always fixing security bugs. That doesn't mean they ONLY do minor updates for security bugs though.

[0] https://support.apple.com/en-us/HT209149#macos10143

[1] https://support.apple.com/en-us/HT208533

None of those were released when the operating system they applied to was a "previous" version of macOS.

Oh ok, I misunderstood you... sorry

I am the author. I always do dot upgrades, as they are (usually) security updates. I wouldn't say I am "stuck" on any particular OS; I choose to remain on it for stability.

They have a habit of spamming this request semi-randomly, seemingly without underlying knowledge of an attempted fix - e.g. you get the same request, at the same time, on several bugs. Sometimes it’s obvious it doesn’t come from any knowledge of a fix, e.g. asking to verify if a feature request is still a problem when no such feature was added.

So you say no, it wasn’t (it never is for me) and the bug goes back into limbo.

My “favourite” macOS bug is 14 years old and I have reported it countless times, including in email exchanges with Federighi. The OS doesn’t respect an app’s input language and changes it when switching apps. Default input, English, gets overwritten on app switch. There is an option, it doesn’t really work. This is my only serious problem with the macOS I love and it’s sad we get new bugs galore while old ones are ignored.

I've been bitten by the Spotlight key combination and the input switching key combination being similar, but this is a PEBKAC rather than a bug.

I've since disabled the input switching key combination because it conflicts with Emacs' set-mark-command.

> It’s okay to log bugs that are missing information; developers can ask for specifics of what they need. In general, for AppKit related bugs, I would find that a “sysdiagnose” rarely would provide any useful information. I would hate it when people (usually QA, and not the engineers who worked on the code) would ask me for a sysdiagnose, when I know it wouldn’t help them figure anything out.

Yeah, except Bug Reporter nags you multiple times for a sysdiagnose if you don't attach one (they have been slowing adding these, I've noticed), and if you ignore these screening is going to send it back to you a week later, without reading the bug, asking for you add one. Sometimes if you do have the sysdiagnose they'll ask for something equally stupid (system profile, usually).

If it can be collected, I guess it's a good process for them. It may not be super user-friendly, but if enough bugs require sysdiagnose for solving, then it's an improvement to not even bother devs until it's provided.

It may not be needed then, but it still saves time.

As the article says, it’s almost always useless.

Even if it weren’t, at least some amount of thinking should be involved in requesting it, and e.g. don’t do it on feature requests or API defect problems. But they do, oh they do.

It should also be noted that it doesn’t save time: it off-loads some time wastage to an external developer who is filling the bug. And who has to pointlessly collect it (it takes several minutes and knocks your system off), then upload 200-400MB file to a notoriously crappy form that it will time out or error out, or sometimes say the upload was successful only to get a bitchy QA reply a week later to upload sysdiagnose.

A few years back, I had a sysdiagnose so huge that Radar just wouldn’t take it. I spent an hour trying to upload the damn thing and eventually uploaded it elsewhere and posted an URL.

To this day, that sysdiagnose file was not downloaded a single time.

FWIW, Bug Reporter has not yet required sysdiagnoses for feature requests. And for large sysdiagnoses, you can ask for an alternative method to upload them; at some point I was given credentials for a FTP server I could upload it to.

The sysdiagnose does contain useful information that is useful for certain bugs. If an app crashes, then the crash report is essential. The log file is also (sometimes) useful. It is easier to ask for a sysdiagnose that gathers the kitchen sink instead of the developer just providing the piece needed (ie: the leaky faucet). -corbin

Yeah, I know that, but the screeners will often ask for a sysdiagnose in cases where it's clearly not useful (for example, visual bugs that exist on an out-of-box installation). So it's pretty clear they're either not reading the bug, or going through some sort of checklist that one of the engineers gave them (don't bother me with a bug without a sysdiagnose!)

Yeah, it is stupid. I used to see this all the time.

The other day I reported a bug in Safari's regex engine and they asked for a sysdiagnose.

I just don't have the energy to perform pointless tasks, so their browser can stay broken.

(In case Apple is reading: /.*?x/u.exec('ふx') should match but doesn't.)

At an Agile presentation several years ago, they suggested the following approach:

a) Bugs get broken down into manageable tasks of roughly the same size (eg. about 1 day to fix).

b) You measure the average velocity that your team fixes these triaged bugs.

c) When the number of bug reports exceeds the teams velocity, you know that number of bugs will never be fixed.

d) Be brutal, and flag that number of the lowest priroty bugs that will never be fixed as WONTFIX.

Now you have a manageable queue of bugs, say as many as your team can fix in two months. And the submitters know the score and know waiting isn't going to help. Lots of productivity and efficiency bonuses all around.

But nobody does this or any of the similar variations, because people rather leave bugs that everyone knows will never be fixed in the system 'because they might be' or because they don't want to listen to the butt hurt submitters complain because they place a higher priority on fixing the bug than the people they are trying to convince to fix it.

And yes, this approach might seriously piss customers off enough that it is a PR nightmare, so I'd be interested in anecdotes of this sort of approach being applied and how it worked out (even just internally).

Or just have a zero tolerance for bugs.

That's effectively having a zero tolerance for functionality too.

No it isn’t, in fact you could argue a better quality code base supports faster development.

You have some examples of bug-free software that does useful work?

That's not what I was suggesting and I think you know that.

What are you suggesting? I truly don't know what "zero tolerance for bugs" could mean.

I don't know if you mean fixing every bug, or not creating bugs in the first place, but either way it is just the case where the teams bug fixing velocity is greater or equal to the rate of incoming bugs.

I first saw the suggestion at https://www.joelonsoftware.com/2012/07/09/software-inventory...

He makes a reasonable case that not keeping the bug queue manageable leads to increased cost. I've definitely been in bug triage mtgs where we spend more time discussing a bug than the code change would take.

> Management needs to allow engineers to have more time to do this, which is at the expense of working on features or fixing already screened bugs.

IMO, this sentence is the key. For Apple to improve its software quality, it needs to stop trying to do many other things. We don't need new versions of Mac OS X and iOS every year anymore, each with incremental feature additions and new bugs.

Anecdote: Yeah I was a bit surprised that a kernel/filesystem bug I logged back in November complete with a reproducer didn't even get acknowledged. I hope they'll eventually get to it, as it causes spurious build farm failures for PostgreSQL. My hypothesis was that the lack of appropriate category might have doomed my report to the random/other/who-cares pile.

http://www.openradar.me/radar?id=4970461708419072

It’s prioritized and in the queue...

Thank you!

> This is terrible because many engineers will not look at bugs with a low priority.

So taking inspiration from the React scheduling work, would it make sense to give bugs a timeout based on priority to keep them from sitting around forever? Say you have a low priority bug, it sits around forever while more important stuff gets fixed, but if it's been sitting idle for 6 months it gets an automatic priority bump?

They can't even fix external display issues with their laptop. I would say bugs in their issue tracker that aren't even that high profile are hopeless.

Does this help explain Apple's bizarre inaction after learning about the FaceTime bug?

https://news.ycombinator.com/item?id=19029573

Filed a bug with Apple 6 months ago. Haven't heard anything and the bug is still there.

How to reproduce the bug on iPhone/iPad:

- start playing some music

- open control center and swipe down on the volume to mute it

- turn up the volume one click on the physical buttons

- swipe down on the volume in control center

- repeat turning up the volume with the buttons and muting in the control center until the control center volume stops affecting the actual volume

Yes, I could reproduce this easily.

I haven’t had an empty backlog since 2011.

It seems that Radar queues can only grow.

When I first joined my team, my manager told me to take a screenshot of my queue, because that was the only time it would ever be empty...

“Feedback” on public betas seems to be an even lower priority.

I’ve reported that each iOS beta gnaws away system storage since they started the program.

My phone is currently giving up 176GB to system storage and seems to have no end in sight.

So many enthusiast users I know are so frustrated they want to give up.

Feedback on public betas is probably lower priority than radars, unless a lot of people are reporting something. I'm sure the quality of the bug reports is much lower.

> I would poke around at code and attempt to make theoretical fixes for bugs I couldn’t reproduce.

This sounds like a great way to accrete unnecessary code, some of which will create even more bugs.

I doubt he actually committed the changes to their 'master' branch. I keep a lot of such tiny experimental changes in my local branches.

FWIW, I would check in these fixes. Sometimes they would work, sometimes they won't.

Code to fix problems, even in theory, isn't unnecessary. Statistically, changes I did were very unlikely to produce bugs. I wrote about this in my post about why you shouldn't do code reviews: https://www.corbinstreehouse.com/blog/2018/01/code-reviews-p...

Interesting article!

I worked for 5 years on a team who had just started using code reviews. What we found was that code reviews didn't often find bugs in the changes themselves, but they were very effective at preventing bad decisions that would cause bugs in the future.

There are lots of ways to make a change that gets the job done, and might even fix a bug, but can mean twice as much work to deal with in the future. These range from intentional but shortsighted hacks to simple mistakes by experienced engineers working in a large project.

You also mention style nits. I admit even I get annoyed by these. But bad style can make it harder for new hires to learn the codebase, which slows the team down. In rare cases, they can even result in bugs of their own like the infamous Apple "goto fail" bug.

Every team is different, but I can't imagine working on one without code reviews now.

And this is the foundation they want to build their little new "services" empire on. Good to be a fat cat… until it isn't.

Encountered a pretty major macOS bug recently where some new OS update made it impossible to enable a root user via the regular route. Made it impossible to set up a local/pseudo-distributed hadoop cluster since the main hadoop "start" command requires "sudo su" to work. Doesn't surprise me their bug reporting system sucks considering I couldn't figure out the right place to submit my bug nor could I find a new way to do this

What was wrong, exactly?

Apple have a real quality problem. I would say it is worse than Microsoft Connect.

MS connect was deleted, breaking all old bug links: https://docs.microsoft.com/en-us/collaborate/connect-redirec...

Sorry - worse than it was. And yes, that decision was sheer idiocy.

Each team at MS is independently ran. I worked on teams where all bugs, and I mean all bugs, with more than ~5 repros in the wild, got fixed.

Then I worked on other teams that just gave up. Sometimes for good reason, fixing long standing bugs could mean a major re-architecture of existing code that would likely induce more bugs than it would fix.

But I also saw teams give up due to sheer overwork and lack of time.

Management matters.

They're not even trying until they can match the idiocy of Microsoft's Feedback Hubs though.

Making users their testers AND not giving them good tools to file bugs takes a special kind of hubris.

Reply on news.ycombinator.com