Hacker News Reader

Comment by tomduncalf | Hacker News Reader

I wonder what is going on with software quality and testing at Apple. It feels like recently there have been quite a few issues like this (the FileVault password bug, numerous issues with iOS 11, the issue that totally broke iOS Safari a couple of years ago) which should have been fairly easily caught, especially given the limited range of devices their software runs on.

I know testing is hard, but a company with Apple’s resources shouldn’t be making slip ups like this. It suggests some real issues such as lack of unit/automated tests and/or sufficient release testing, which pretty urgently need addressing.

Anyone got any inside scoop?

View on HN

macOS and iOS updates at Apple are now inextricably tied to new iPhone releases. There is a strict yearly deadline that the teams sprint toward, a timeline imposed by marketing rather than readiness. This affects prioritization of which features are pursued, where they lie in the stack, and how polished they get.

Insufficient testing at today's Apple is not limited to software. They bragged about their extensive input testing lab [0] when the new line of Magic accessories was released, but the Magic Keyboard with Numeric Keypad launched last summer had all of its inventory pulled from the channel last month because users discovered that the model was so thin that its midsection bowed over time.

[0]: https://medium.com/backchannel/what-i-saw-inside-apple-s-top...

Everything ends in tears when managers mix up targets/estimates with commitments

it is also that they pursue features just for the sake of it. things get moved arund in the iPad from release to release for no good reason, often going backwards in usability. every release i have to relearn simple things like how to manage the screen brightness. i really wonder what they are thinking internally other than “we need to shake things up to make it appear we’re doing something with stale products”.

It seems phones and tablets have reached the stage where laptops were maybe 15 years ago. All the major features are done and innovation is pretty much over. So they have to make a lot of cosmetic changes that look like activity.

Where did you hear that the keyboard was pulled from the channel? It just seems to be out of stock for me.

Haven't deadlines at Apple always been driven by marketing? I'm looking for a source but I remember a story where the product director for iPod was told by steve jobs "make it simple, fast, beautiful, and have it done by Christmas."

That's sure to send shivers down the spine of anyone reading it here but, to be fair to jobs, he managed to get exactly what he wanted on that occasion.

Asking for stuff requires no talent, vision, discipline or effort whatsoever. Pretty much anyone can do it. If you don't actually deliver, you don't actually matter.

Take this for the anecdata that it is. I interviewed at Apple, referred by old Microsoft friends that worked there. As I was trying to get a feel for things before the interview, I asked about the software testing. I was told, "don't expect what you're used to at Microsoft". The reference there is from when Microsoft often had more testers on a team than devs (ah, the good ol' days). The summary of what I was told by friends, and the questions I asked during the interview, is that testers at Apple aren't the testers that Microsoft used to have. Microsoft had testers working in MS Research, researching ways to better test software. Apple, from the impressions I got, is doing good to have testers than can write "hello, world". This was from the app side of things, not OS; I don't know if it's any different on the OS side.

But since I don't work there, I have no good inside info. But just from gut feel, I don't think my anecdata is too far off the mark. Based just on the bugs made public, I just don't get the impression that there are testers at Apple whose sole reason for being there is to tear into a piece of software and break it. There was a bug a few weeks ago posted to HN that I commented on. I don't have a link without digging through my comments, but it was something along the lines of "how could a tester not find this in five minutes of exploratory testing?" This bug is similar. It would take more than five minutes, but were this my area to test I'd pick at it once in a while when I had a few minutes. As I pick at it, I wouldn't expect to find anything, but I've got a minute between builds, so instead of randomly clicking Facebook I'll randomly click this dialog. What did the dev forget? What weird state was not accounted for? Some kind of state overflow if I click the button enough times? Shove some Unicode in there, that didn't find anything; meh, maybe I ought to move o...hey, wait a minute. Did that thing just log me in as root?

But my gut says that Apple doesn't employ a lot of testers like that.

> But since I don't work there, I have no good inside info

Actually, I've been wondering why I hear less about people working at Apple than at other big tech companies. It seems everyone and their mother work at Google or Facebook, but no so much at Apple. Do they have less software engineers, or their employees are required to be more discrete?

> or their employees are required to be more discrete?

Yes, I believe so. I've heard there are strict requirements on even internal discussion. (Who you can talk to; about what; where.)

I believe there was an article about how Apple was finding it extremely difficult to hire ML experts because of the secrecy they require. In order to mitigate that they mad an exception for their ML/AI engineers allowing them to publish papers to external journals and present at conferences.

Presumably this is why their software is increasingly shoddy. But I wonder whether it's a direct effect of poor internal communications, or an indirect effect where the ludicrous secrecy has driven away any half-decent programmers.

Apple probably doesn't take too kindly to their employees talking about their work. I'd imagine it's a fire-able offense.

it is everywhere else too, but people do quit...

Could it be the level of secrecy around Apple? I see responses for Google and Facebook devs on HN a lot but never Apple.

The only people I know locally that work for Apple are remote customer support folks.

Do they have less software engineers, or their employees are required to be more discrete?

I know but a few that work at Apple, and of those few they strike me as less forthcoming than the multitudes I've worked with and know at Microsoft. I've wondered if part of that is because Microsoft previews/pre-announces just about everything, whereas Apple (mostly, and not so much anymore) announces it when the shipping trucks show up at the local Apple store.

So the outcome from the Microsoftie is, "it'll do this that and the other, but that's all I can say right now." From a recent conversation with an Apple employee: "they make me go in a special room to use the hardware, and I can't work from home. That's all I can say."

Probably more so, last I looked, Apple has considerably fewer software employees than the other big companies.

> Probably more so, last I looked, Apple has considerably fewer software employees than the other big companies.

I don't think this is true. Apple, Google, and Microsoft all have on the order of 100K employees.

Keep in mind that Apple directly employs retail staff.

Point taken. A closer look shows that about half of them are part of Apple corporate.

As a Tester myself, I cannot understand why this is not covered by either unit tests or behavioral tests. Clicking dialog buttons in rapid succession is what we (should) do once in a while. Especially in core functionalities such as the login screen. It's one of the first screens you see as a tester. And you have default usernames, be it enabled or not.

For example, I do not own an iPhone, but at work, I made a bet with my colleague (jokingly) that I could break _something_ on his phone in a few minutes.

I did not have his finger print or pin-code, so I was very limited, I even joked "I don't need that, give it here!"

Finding out I only had a hand full of options, I focused on the emergency dialer. As any good tester would be curious about, I wanted to check the max field length, so I entered digits, copy/paste it a few times, copy/paste that string, ("wait, no limit? Not even at 1000? why?") and so on, until I noticed the interface became laggy, so of course, I kept going.

Boom, suddenly back at the login screen, tried to open the emergency dialer, but got a full blank white screen, in the meantime the phone started heating up substantially. Since it was a new Phone (iPhone 7 with iOS 10.x I believe) and the dev getting nervous, we decided to reboot it. That fixed the issue. (Curious if this is still an issue in iOS 11.x)

TL;DR: As a tester this simple curiosity should be in your blood, and especially covered in behavioral tests when your software has been around for 5+ years.

This bug isn't caused by rapid succession or whatever, it's more of a generic end to end test. In this case someone would have had to write an exact scenario that opens a settings page, unlocks it, types in 'root', no password, presses login and it should not work.

In this case, any password works. The blank one is just the initial example. The password is set to whatever you typed, even if nothing.

Rapid succession button clicking usually combines many different tests:

- Performance

- Usability

- Brute-force capabilities

- Error handling

- And in this case, Security, a bug where trying to log in a couple of times on the Login screen with an empty, or set, password.

A test scenario closer to this would be:

---

When I am on the login screen

And I enter 'root' in the 'Username' field

And I enter 'thispasswordisfalse' in the 'Password' field

And I press the 'Login' button '10' times

Then I should see the text 'Your password is invalid'

---

Please note that this issue is not just in the Settings page, it takes place on the login screen as well, that's why I'm shocked, it's such a core functionality, touching so many system components.

I got my friends Apple Watch stuck last week just by looking at it's features. IIRC it got stuck while using the "flashlight". It suddenly froze, and it took me a while to reboot it (it got stuck once more while rebooting).

All in all, it took me about a minute to break it, and around 5 minutes to get it working again. I was getting a bit nervous.

Unrelated to Mac OS but I used to wonder all the time why iTunes connect was so shoddy. I got my answer when I learned Apple had outsourced a ton of backend work including iTunes Connect, App Store backend to Infosys in India.

They’re now retreating from that strategy: https://factordaily.com/apple-to-pull-back-development-work-...

well that explains allot, iTunes store/ App Store have always been painfully slow, compared to doing anything in Safari.

Everyone retreats from that strategy.

It seems apple's software has been trending down in quality since Snow Leopard.

I'll agree that Snow Leopard is the high water-mark.

I have been in the apple ecosystem for about 10 years. For a company that has been priding itself on end user security, the bugs that have been creeping their way into the OS are just... disappointing. What is the point of paying a premium for a well polished hardware/software bundle if the OS is malfunctioning in a non trivial manner. Design? Right now when I use my calculator app on my iPhone and do 2+2+2 I get 24. That's a pretty awful design. Actually, it's a lie.

3rded. I wish time stopped at 10.6.8.

It can't be a coincidence that this was the last major release before Steve Jobs died.

It can’t be a coincidence because it’s not true. Lion was released well before SJ’s death.

Unless you’re arguing Lion wasn’t major because you didn’t like it, but that’s an argument that proves to much, methinks.

It's common among a small group of Mac users to hold Snow Leopard up as the peak of software quality, but only because of rose-colored glasses [1].

[1] https://www.computerworld.com/article/2528936/mac-os-x/snow-...

I should have specified I meant 10.6.8 I ran it on my main computer until 10.9.2 because of the problems I had experienced with Lion and Mountain Lion on other computers.

Also I think when most people think of Snow Leopard they're thinking of 10.6.8, at least that's the version number you always see get thrown around on the internet.

The last High Sierra update will have less bugs than the next major release too, so I don't think that's really relevant.

Your right but I think the biggest Issue is Apple does much more frequent releases now than when they were doing Snow Leopard. So more iteration hence more stable OS.

It's the last release they made any significant updates to the BSD userland. I'd mark this as the release they ceased serious investment into the operating system itself. After this point it's almost nothing of note. If you look at the dates of the various tools etc., this release is when they were last updated. Many are now getting on for being a decade out of date. The only major change has been the switch to llvm, and they made that horribly painful.

It also marks the decline of the desktop UI to introduce increasing amounts of iOS-like behaviour and appearance to the detriment of a usable desktop. Like proper scrollbars etc.

While some nostalgia might account for holdouts, it was the peak of MacOS in the minds of many, including myself. As a developer, I've been quite disappointed by its direction and declining quality. For the amount we pay for this hardware, it's not much to ask for some basic maintenance work and testing to be done.

I remember my Macbook's battery life being significantly extended by an OS upgrade; either Leopard or Snow Leopard.

There's been releases ever since that improve battery life, or, keep it the same while doing more; background apps get throttled, graphics rendering has been optimized using Metal, and in future Mac devices, I'm sure they'll put their mobile processors in it to handle background tasks - it might even be strong enough to power the next generation of Macbook Air.

Mavericks was a big one for extending battery life. It came quite a bit after Snow LeopRd and introduced timer coalescing which significantly improved battery life across the board for all MacBooks:

https://m.imore.com/mavericks-preview-timer-coalescing

Apple has always had QA issues, the difference now is that they’re increasingly tested by the users, hackers, etc.

Difference? MacOS userbase hasn’t changed much since 2011, I thought?

I've no information on how good this site's data is, but https://www.statista.com/statistics/218089/global-market-sha... seems to show that from 2013-2017 global macOS market share has increased from 7.95% to 11.3%.

The loss of people like Avie Tevanian and Bertrand Serlet took its toll.

Are there any "(tech) household name" engineers doing system-level work on iOS/macOS these days? It seems like Google and Facebook have a slew of them.

In the case of Facebook, all the “household named” developers do nothing to improve the quality of their output, be it their user-facing software or developer-facing open source.

Dominic Giampaolo of BeOS/BeFS fame. He now works on APFS at Apple. Their work is really - impressive APFS was announced in June 2016 and rolled out on iOS devices in March 2017. Given that the APFS roll-out was relatively uneventful and how they tested it [1], it seems that they can still do low-level engineering and proper testing.

Of course, until recently they had Chris Lattner as well.

[1] For some iOS releases, they converted HFS to APFS in-place, report the results back to Apple, but did not write the APFS 'superblock' to keep the filesystem HFS+. It's quite a smart idea, because they got reports from millions of devices without actually switching them to APFS.

Whilst APFS is an improvement and I fondly remember using BeOS back in the day, I am not sure APFS is that impressive feature-wise compared to NTFS. It's still miles behind NTFS, which is now ancient.

Repressive collusion to fix salaries and restrict industry movement, doesn't really inspire your employees to try their best

> Anyone got any inside scoop?

I have a feeling that anyone who does would get fired for commenting here about it.

You have to think that whoever was responsible for testing this is going to get fired. This is egregiously bad...

This is a management issue. It shouldn't be possible for such a mistake to slip into production code. It has happened more than once in recent times.

iTunes had QA problems for more than 10 years, only the early versions were really solid. I am not sure that it is a recent problem.

Subjectively it feels like Apple bugs have become larger and more prevalent, over the last few years. That and IMO clean OSX/iOS installs don't quite feel as polished as they used to. (I stopped using Apple products, except for a MBP, for a few years and recently started using them again, and the MBP still runs 10.10 for precisely this reason) The last solid OSX release was Snow Leopard

They've added features over the years without removing or polishing them; there's Launchpad which was added in a period where OSX seemed to lean towards becoming touch-friendly, but it didn't replace any existing feature (iirc) and just feels off. Might just be me though. Notification center? Don't use it.

A [?] don't know what you are talking about.

I've experienced lots of issues wiht IOS and High Sierra as well. Apple definitely reduced their software quality since IOS 11.

And seeing this I am wondering why people still trust closed-source software. My long term dream is using 100% free software on a HW with minimum binary blobs.

This also can happen in open software. So I don't think your comment is valid.

Open software enables people to take a look inside to what is going on. It isn't a cure for bug free development.

It reminds me the KMail bug: https://www.ctrl.blog/entry/kmail-cve-2017-9604-openpgp

Some security bugs exist in the Linux/BSDs kernels for a loooong time before someone notice and fix it (e.g., https://media.defcon.org/DEF%20CON%2025/DEF%20CON%2025%20pre...)

Anecdotal but I started noticing a decline in quality after Steve Jobs died.

i'm still wondering if that impression ( which i share) is real or not.

At the minimum, i'd say i feel apple release less innovating os versions while producing at least the same number of bugs.

Reply on news.ycombinator.com