Apple Sent Two Men to My House | Hacker News Reader

Apple Sent Two Men to My House

blog.vellumatlanta.com

254 pointsglhaynes101 comments

Open articleView on HN

Context[0], since I didn't see a link in the article. Also, the discussion[1].

[0]: https://blog.vellumatlanta.com/2016/05/04/apple-stole-my-mus...

[1]: https://news.ycombinator.com/item?id=11634600

Thanks. I think perhaps the article should link to some context...

He's making the most of his 15 minutes. Viz: https://twitter.com/settern/status/728344954171437059

This whole saga has been a pretty interested set of events, amusing to watch as a bystander who's been observing Apple for a long time while they go through many changes.

Regardless, reminds me of Eddy Cue driving over to Federighi's house late one night to report a bug[0]:

    When Cue ran into a problem installing a new build of OS X on that iMac,
    in fact, he could tell as a veteran software tester that the bug might
    be hard to reproduce, plus he was scheduled to take a trip the very
    next day. “I called Craig up, said have your guys look at it, I think it
    would be hard to re-create. He said sure, so I put the iMac in my car
    and drove it over,” as in, to Federighi’s house. Cue went on his
    business trip, Federighi’s team fixed the problem, and Cue got his iMac
    back when he returned—kind of like a Genius Bar for the C-suite.

[0]: http://www.macworld.com/article/3033057/ios/eddy-cue-and-cra...

It's great that this happened. It's annoying that it indicates the best way to get support from Apple is to make a prominent blog post.

> It's annoying that it indicates the best way to get support from Apple is to make a prominent blog post.

Posting somewhere public (for example, a company's social media page) to get support appears to be an increasing trend.

Which really means "we don't care if you're unhappy, we care if other people know you're unhappy" :(

Maybe instead of thinking of it as some sort of decline in service we should consider it as a net improvement in global communication. Media has democratized and you don't even need to convince the local TV news crew to cover your story anymore.

You can just make a clear, coherent complaint in your own forum and occasionally [like in this case] it actually brings about a transformation.

I doubt that Apple's customer service is any worse now than it was before.

I'd like to think that is the case... When I was a child, we got a phone number that was one digit away from the number of a major hospital. That would have been bad enough, but Norway had area codes that covered very small areas at the time, and were optional within the area. Lots of people didn't remember the area code and so always used it, but would guess based on location of the person they dialled. The one digit difference was in the area code..

As a result we started getting calls about medical emergencies in the middle of the night.

The local phone company did not see this as a problem either for us or the hospital or people calling with emergencies, but offered to put us on a waiting list (!) potentially for months before they could be bothered to change the number.

Until my dad had hounded the reporters at the local newspaper until they wrote a piece about it - suddenly it was fixed within a day or two.

So public attention worked back then too (this was early 80's) - there was just fewer outlets and more work to get your problems published.

These days it doesn't take much before I go for social media.

Another option I've found works (but is pricey) is to sign up (temporarily if you don't otherwise need it) for one of LinkedIn's premium tiers and use that to get access to top execs at the company you have an issue with. Often the issue is not so much to draw public attention, but that drawing public attention is the simplest way of getting the attention of and access to a senior enough executive.

> Often the issue is not so much to draw public attention, but that drawing public attention is the simplest way of getting the attention of and access to a senior enough executive.

I really believe that's true and wish it could be applied in more cases. Basically spread the problem to people who are normally not affected by it.

I would bet that TSA queues would get solved in ~2 months if all government workers and politicians (to the highest position) were excluded from TSA precheck and could not pay for priority queues.

That's what spurred me to try LinkedIn when DHL messed me about - I'd had success getting issues solved when senior people happened to have their attention brought to my blog or whinin on Twitter, so I figured I'd try removing the middleman, at the cost of spending money instead. It was an interesting experience (especially as I got cc'd on some very interesting e-mails where people where told to fix my problem...). But that was a relatively easy situation that "just" required the attention of a couple of SVPs. Finding ways of drawing attention to more serious issues is a lot trickier.

> You can just make a clear, coherent complaint in your own forum and occasionally [like in this case] it actually brings about a transformation.

But only if you have enough followers or readership.

The Ars Technica writers have often remarked about feeling guilty for taking advantage of this privilege. They try to resolve things, then after failing, mention they're going to write a story about things, and suddenly get the VIP treatment. But not everyone can write an article for a major news site.

But you can tag their Twitter account.

Does this always work? Is it going to be somebody's job to monitor all tagged tweets?

I wonder if it's part of a longer-term strategy to get people to ignore online complaints altogether. At some point, when it becomes commonplace to tweet about problems instead of calling, it'll be substantially more difficult for horror stories to "go viral".

The more likely explanation is that PR teams are horrified at the prospect of something "going viral" and don't know how to tell what will and what won't, so they work to resolve any potential issue before it can blow up.

I'd say it means that the social team monitoring twitter has more power to get shit done* than the CSRs manning the phones.

*in this case meaning escalate to product teams or get the attention of executives

Isn't that the same though? If you people who listen to you privately can do less than people who listen to your public statements, I think it goes back to my original argument. The company empowers people who can stop bad news rather than those who can make you happy.

Or maybe: we care if it registers to our radar, because else we get tens of thousands of unhappy people complaining to us everyday -- and most are just ignorant of how it works, unhappy for no reason, have installed all kinds of crap and done custom changes that broke their system, etc.

Hasn't this always been the case? It's why the Better Business Bureau exists.

"Reaching out via social" seems to be the new "can I speak to a supervisor".

True story - a month ago my credit card bill showed a mysterious $100 charge from Amazon. Phone support found the charge but couldn't figure out why it had been made, and eventually asked me to call my bank and dispute it(!!). That sounded like crazy talk, but rather than arguing I just said thanks and tweeted @amazon. Their social people referred me to someone who wound up figuring out how to cancel the charge.

Disputing the charge is the right thing to do in any case. Your CC company should make it very easy to login and mark the charge as problematic. They want to know if bad charges are coming in from the vendor, and this is the best way to track it.

I personally don't think twice about disputing fraudulent or even mistaken charges. I can see how you might want to reserve this tool as an all-else-fails, but really CC companies should be appreciative of people using the tool because it gives them useful signals. Much better than the many many others who are missing the charge and paying for it by mistake.

Disputing a charge just because you don't immediately know what it is, is not a good idea. At least, not if you want to continue doing business with the vendor that made the charge. Companies tend to take a dim view of people issuing chargebacks.

That's an interesting perspective. I take a dim view of companies charging my card incorrectly. Particularly Amazon since they submit many separate charges for items within a a single order, making it virtually impossible to verify the charges.

I believe it is against the merchant agreement to withhold service after a valid charge-back. That would be a major and unwarranted retaliation, which you could probably sue and win a large class action if this is really happening.

I will say, I should have made more clear, the first step in a dispute will be to contact the merchant. They will have a fixed number of days to reply and correct the problem.

There was a somewhat recent case of someone having their amazon account with all their media they purchased licenses to view via their Amazon account taken away from them permanently for issuing a charge back due to some disputed activity on their account. So, before you go around issuing charge backs, you should think of the consequences of disputing your account status with that merchant. You may end up losing access to ALL your Steam games just because some kid cracked your password and purchased a $1 game with your account. This is the danger of having an account on these digital distribution sites everyone seems to be so fond of. Sure, you can download your movies without having to leave your house to go to Blockbuster, but what do you do when you get your entire account locked on the Microsoft Zune Marketplace because of one charge? You could lose your entire iTunes library because you failed to recognize one charge that your kid made without your permission. Any such event could (and very well may) actually cut you off from your entire media library while you resolve your situation, if you actually manage to solve it. What if someone cracks your Battle.NET account and uses your copy of World of WarCraft to run a gold farming bot or a cheating script, and now you lose access to your entire Diablo 3 Real Money Auction House funds. You could literally have thousands of dollars taken away from you without it being your fault in the slightest. You know, provided you have a time machine and go back to 2013 when the D3 RMAH was actually a thing. In any case, this is a real danger where you can lose a lot of value over a simple charge back, and there is little to protect you from this retaliatory account locking that distributors of digital goods are so fond of lording over us plebs. Again, this is a serious issue. Do consider it whenever you think about issuing a charge back on your accounts. Now, the more faithful a customer to a digital distributor you are, the more you get screwed over by complaining about them mistakenly charging you for things you didn't purchase, or other such situations where you are forced to issue a charge back because you couldn't get the vendor to resolve an issue with the money they charge you for transferring some data to your system. I won't even touch the discussion on how physical media cost the same as the digital access despite the obvious lack of printing the medium, producing it, shipping it, warehousing it, displaying it in a store, securing it in the store, advertising it in the store, and selling it via a cashier. Running the download and billing servers has to be several orders of magnitude less expensive, not to mention that some distributors (Blizzard) use peer to peer downloads to make you distribute the software for them in addition to downloading it. And somehow they are the same price. Go figure.

ccrush, this is a good comment, but I found it really hard hard to read due to the lack of paragraph breaks.

You can add breaks when commenting here by leaving a blank line (double enter) between paras.

Apologies if you knew this already.

Losing my Steam account due to a chargeback is something that never occurred to me. It's a chilling thought!

My original comment actually included a bit about how common knowledge is that Valve can and does lock you out of your Steam account if you issue a chargeback, but I decided not to include it because it's really just hearsay (I don't know anyone personally who has experienced this).

Witness the wailing when people get locked out of their Google accounts and lose access to email, Gdrive, photos, Google Analytics etc.

Same thing can happen with any account, of course, but Google users tend to have a larger number of more important eggs in a single basket....

Seconded. On the rare occasion that I will dispute a charge, I am one hundred percent sure I did not initiate it before contacting my card issuer. If I think there was an error rather than fraud, I'll always try to work it out with the merchant first rather than risk my welcome to do business with them in the future.

FWIW, in principle I agree with both points being made, but in practice I just didn't want to be responsible for following the issue and making sure it got resolved correctly. I wanted to hear "oops, leave it to us" and then stop thinking about it.

Perhaps worth adding, I was/am 99% sure how the charge occurred, and that it was a Heisenbug from when I'd trialled Prime. So there was no question of punishing them for being a bad vendor, I just needed them to bounce the issue around until it landed on someone who could fix it (which is essentially what happened).

I think the real root cause of situations like this is that social teams sometimes have more/better options for escalating issues than customer support does.

That was not the case here though. Amazon probably said to dispute it because that is the way to get it to the right people. The advantage of that from your prospective is the CC company takes over with dealing with Amazon for you and it's off your shoulders.

Yes, but I wasn't responding to fenomas, I was responding to zaroth who said

> Disputing the charge is the right thing to do in any case.

You are misguided in this. For the vendor/service provider/whatever, it's a $15 charge per chargeback regardless of the outcome. So If I sell you a monthly service for $6 per month and you forget to cancel it after three months, then at end of year check your statement and notice the charges and decide to make chargebacks to correct it...Then you just cost me 96 + 915, $189, which means I am in the hole $171 because you suck at managing your checkbook.

This is bordering on ranting, but happens all the time and is just a cost of doing business, which means my regular customers get to pay more to deal with people like you. Contact the merchant first, they are usually very prompt and want to avoid chargebacks like the plague, because they cost real money.

I'm sure you're right about the probability that many small business owners would still behave honorably if problems were brought to their attention first, but GP was, verbatim, referring to "bad charges coming in from the vendor," and "fraudulent or even mistaken charges," which is quite different than the negligent-customers scenario you're describing.

Edit: Not to mention that GP is referring to an anecdote of GGP's in which Amazon told him to dispute the charge because they couldn't sort it out. That's disgraceful.

Yeah, you could say I used the comment as a soap box to rant about something that I experience all the time :) In some cases chargebacks work great and should be used. I've honestly never had to use one, since merely mentioning it to any merchant or service has been enough to get my problem resolved promptly. Sorry for the derailment :)

I completely agree that subscription charges for a service you signed up for and could have, but didn't bother to cancel is not a valid chargeback. If you initiate one and complete the interview questions honestly I think the chargeback would actually not be allowed. (Do you recognize the merchant? Yes. Did you approve the charge? Yes. Did the merchant deliver the product or service? Yes. Hmmm... Sorry, we can't refund your money)

However there is a slippery slope. Two examples that hit me recently, an iDrive renewal where they refused to tell me the renewal price and their online form provided no way to opt-out of the renewal, their online chat told me to call a phone number, and it turns out the price was 10x the first year cost (and not at all competitive with any other backup service). Another was BitTorrent Sync which was supposed to be converted to a one-time license but then lo-and-behold a year later another charge showed up.

But for merchants that provide the ability to cancel a subscription online, and who let users know before an annual charge is about to hit your card, I agree the customer has some responsibility to manage their subscription.

My personal opinion is that monthly recurring billing can send invoices after the charge or even no email at all if charges are the same amount every month. Annual recurring should email a week before the charge to remind a customer it's coming, or should allow a full refund of the charge if contacted within 30 days of it hitting. For example LinkedIn refunded an annual recurring charge that I contacted them about within a few days of it hitting the card for a "subscription" I didn't actually want and it just took a single email to sort out.

In any case, if I signed up online I must be able to cancel online with a click. I won't play games with merchants who have dark patterns that make me jump through hoops or pick up a phone to manage an online subscription. These merchants deserve the extra chargebacks they get, and they know exactly how they could reduce their chargeback rates. They've just done the math and found the dark patterns unfortunately are net-profitable.

Exactly. Whats amazing is I was able to get the DMV of all places to help me out. https://twitter.com/CA_DMV/status/698272726788632576

They ended up calling me to make sure everything was right, and called back a few weeks later to make sure I got my ID. Never had to step foot in an office like their phone support said I would.

Annoying but true. I'm not sure if it's because they're more worried about bad PR, or if the number of people complaining on Twitter is smaller than the number complaining via formal channels (and hence can be taken more seriously).. or if they're considered more technically advanced and hence their problems are more likely to be "legitimate"... or what.

I'd say it's a natural result of structure. Social teams can be small and internal because they can choose where to spend their time - they probably ignore 99% of the tweets they see ("Customer support can help you with that. Thanks!"). Hence they can focus on stuff that actually needs escalation, and being internal they're more likely to have met somebody who'd want to know about an unusual issue, etc.

Whereas phone support is the opposite - it's designed to be handle the 99% bulk of requests that you wouldn't want to be escalated, so it's not too surprising that it's not as good at handling outliers.

My bank (NatWest) kept sending me my bank statements in Braille, even though I had opted out of paper statements completely. Six months of calling them up on the phone or going into branch; each time I was told it was sorted, only for the next month another braille statement to arrive. Complained about it on twitter and within four hours it was fixed.

They don't seem to care unless there's a chance of other people noticing.

I don't think it's malice, or at least as cynical as you make it. It might as well be as simple as the social customer service department being younger and thus not having been subject to the iron law of bureaucracy for so long. Their budget possibly comes out of marketing, not customer service, and thus isn't subject to being considered a pure cost centre. Social media is sexy, when someone from that department makes a call, they are more likely to reach someone who think it's fun to dig into an issue that when someone from the phone bank (to the extent they're encouraged or even allowed to) reaches out. Finally, both volume and S/N of social media pings are much better for many firms as long as the people contacting you through those channels tend towards being young and savvy, but that will change quickly as people catch on, and the canned "Oh I'm sorry to hear that, please call our customer service department on ...." responses that we're already starting to see will become much more prevalent.

When thinking about customer service in big companies, it's important to remember that the vast majority of calls legitimately are of the sort that only require a very simple action to be taken, and then to be disconnected as quickly as possible. However frustrating, it's not irrational hatred of customers that makes it difficult to break through that assumption when you do have a more complicated issue.

Nah, i'm more cynical. I had to complain publicly about ResearchGate to get off their spam email lists. It worked. Nothing else worked.

That's like telling someone upset about a cold meal not to blame the waiter because it could be the cook's fault. Who's at fault is not at issue. What's at issue is that the customer can only get problems solved when the company's reputation is at stake, which is a failure of integrity.

> They don't seem to care unless there's a chance of other people noticing.

Many years ago I was having trouble getting some accounts closed with NatWest (after a few cockups on their part already I moved to FirstDirect). I happen to work for a company that produces compliance software and offered related consultancy so we had copies of the FSA (then new, since reformed as the FCA) complaints manual lying around.

After months trying to get traction on the closing the accounts (including warnings of charges because one of them was "under funded" by virtue of having been emptied because I wanted it closed) including a couple of face-to-face meetings in the branch (back then little was done remotely), everything magically sorted itself out after the day I went in and read from that manual while waiting to be seen...

It isn't just "others noticing": any sort of fallout gets things more noticed.

In hindsight I had a case to just go strait to the authority about it and probably should have done. The bank would probably have got a five grand fine and I might have been compensated for my wasted time...

> They don't seem to care unless there's a chance of other people noticing.

Which is one reason why tweeting works well (or it does if you have enough followers). I've had great results that way from several firms, though I have not tweeted to Apple.

As a counterexample, I've gotten great service from very large companies by sending them private communication via social media (Facebook messages, for instance) - mostly because I really don't want to fill up my social media feeds with whiny complaints.

I mostly do it because it saves me driving in and/or waiting on hold, but those messages usually seem to be answered by the same team (and are generally handled very well).

I've had better response from @amazon than contacting them directly. My normal method now is to post first, then contact them directly. Direct contact has become their Tier 0 support.

This sucks... from what I'm reading here (both your post and a bunch of replies), I'll have to sign up for a Twitter account to get decent customer service in the future.

Apple is not alone. I was an early ouya backer, and when mine arrived, it was water damaged (looked like it had been under water, not just rained on). The usual channels netted no response. I tweeted a photo with the text "I asked for shipping, not submarining", and got action within hours, and a replacement in days. Sad, but true.

As an IT guy more and more often one of my roles is just to deal with the support people of a vendor so our clients don't have to.

The larger a company is the harder it is to get anyone on the phone that gives a damn about your issue. Even better are companies that don't handle their own support, such as Polycom, who will kindly tell you to get bent and call some other company even after you've paid over 1k in renewal costs for a support contract.

I've blogged or tweeted about HP, Ruckus, SonicWALL, Microsoft and probably some others. In each case I at least got to speak to someone that wasn't in a call center. At that point its usually some bizarro issue.

For instance with HP they kept sending onsite techs to fix an issue who were literally lying about going out and fixing something - they weren't showing up. Or with Ruckus they wouldn't stop sending us replacement wireless access points and then couldn't figure out how we were supposed to send them back.

Not surprising that it's more effective than calling a phone support line that's typically at least two layers of outsourcing away from the company that you're trying to contact - and usually to some sort of high-churn, low-trained call center sweatshop[1].

The only thing less effective would possibly be trying to get help from a Microsoft/MSDN forum - "See this completely unrelated KB article for another product that doesn't work. Closed as resolved."

[1] I write some software that gets used in call-centers... It's exactly as much fun dealing with them as your customers as when you are their customer.

The other day I played one of my "Jedi Mind Tricks" albums on iTunes (yes this happened in 2016, because reasons) and as each song played it disappeared from the album list. Apparently the songs were being repopulated in another album under "Army of the Pharaohs" because either iTunes (or I?) had decided to rename the artist at some point.

Probably related, possibly dated: you might want your albums from "The Dwarves" and "Dwarves" (don't judge me man) to be listed together because they are the same band and it's not my fault ITMS has it wrong but if you rename one, good old iCloud will happily download a new copy of the one you renamed.

The Dwarves. I just. I. Who. Why?

I honestly never thought I would talk to someone who would actually own and listen to their albums. Why would you do that? Are you a masochist? The 'music' is just awful, the shows are (at least they used to be) amazing.

But the 'music' is just so bad. Why do you do that to yourself?

Isn't that basically all punk, though? Inasmuch as punk still exists, that is. If you just buy the albums, you're simultaneously engaging in self-abuse and subsidizing the people smart enough to go to the shows, where actual enjoyment is to be had. And, to the extent you let on that this is what you do, you're identifying yourself as an oblivious trend-follower who confuses the trappings of someone else's broken idea of "cool" with anything that actually matters - the musical-taste equivalent of The Office's Michael Scott.

Or have I missed something here?

> Why would you do that?

Nostalgia?

> Are you a masochist?

I hope not.

> The 'music' is just awful,

Some is sure but on the whole melodic/surf punk is far from the worst genre, I've heard a lot worse and I only listen to certain tracks.

Also I gave explicit instructions not to judge me, man :)

>"The Dwarves" and "Dwarves" (don't judge me man)

Hey I enjoyed these guys, never heard them before; thanks.

About 20 years ago, when MASM was still for sale on the retail shelf, I called Microsoft with a problem when the help files wouldn't install (from floppy). The person on the other end showed me how to extract them manually, then later in the week I had not one, but two calls back on my answering machine following up and hoping everything was going okay with my installation and if I had any issues to please call them back. Those were the days.

Aye.

Around the same time a friend of mine sent a physical letter to Microsoft reporting some minor bug with DOS 3.x. Not a couple of months after they replied thanking for the report, confirming the bug and saying that it was now fixed. On an official letterhead with watermark, signed by a real person.

You can get that kind of support (but in email over paper) for many smalltime webapps now. Just doesn't scale well for a million customers ;)

This was originally from an article titles "Apple stole my music", whereas iTunes deleted music from his computer.

I doubt the old Apple would have gone to so much trouble, good to see the new Apple appearing to be more concerned and open.

Probably trying to find a solution for the PR issue

A couple years ago, Amazon deleted Animal Farm and 1984 from customers' devices ;)

All vendors are equal, but some vendors are more equal than others.

Yeah but it was intentional while this ITMS issue seems to be because of a bug.

No, Apple was deleting stuff intentionally. That's clear from the articles and cited Apple boilerplate. The bug just caused unintended deletion.

My point is that any deletion is dangerous and unwarranted. If for no other reason, because it increases the risk of catastrophic bugs like this one. Sure, people should be backing up their stuff. But in this case, Apple acted like filecrypter malware. Worse, in fact, because there's no recovery key.

You don't have to tell me. I don't trust any streaming service. Whatever streaming I have to do I do it from own "cloud".

The deletion thing has been known for months. Now Apple does something?

And a special version of itunes? Does nobody over there collect mp3s?

What would you rather them do? Folks can't have it both ways. In one corner, people bitch because apple isn't doing anything, and in the other, they start bitching because they did do something.

This stuff gets old after awhile. The older I get the more I hate the Internet.

Me too. How about Apple not fucks up in the first place? How about people put out software that works and not this MVP bullshit that passes for a finished product? How about respecting the user by default?

Starting from a position of "Don't make mistakes" from a series of fallible entities (Computers, People) seems like a position doomed to failure.

Make less mistakes? Test more? Don't move so fast?

And you know what? I will expect perfection. Because if someone has the gall to destroy people's stuff they better execute to fucking perfection

Reached back for the bottle, rubbed against the lamp / Genie came out smiling like some Eastern tramp.

"OK, iTunes is now written to DoD/NASA/JPL coding standards. It has zero bugs. That will be $69,105,000.00 per user, please.

Oh, and we expect to support 16-bit audio by 2024. You're good with 8 for now, right?"

So take my comments to the other extreme? Why can nobody see any middle ground??

The pushback experienced for wanting quality, well-written, doesn't-need-to-be-patched code is mystifying

I think thats mostly an illusion, and the costs for that "last mile" of quality is huge.

Think of it this way, if the software works for 99.9% of people, that's still 800k people that can experience undesired behavior. You hear "everyone has this problem!" but in aggregate, its not. Yet 800k is still a lot of people! Yet its 0.1% of iOS devices sold!

In short, what you are asking for is easy to say with words, but in reality its pretty hard to do, even if you have "more money than a nation" or whatever comparison people are using.

You're not wrong, definitely. It's just that criticizing iTunes feels like picking on the mentally-challenged. As rarepostinlurkr points out, even the "best of best" practices can't really guarantee bug-free software. And Apple is very far down indeed on any list of respected software development institutions.

The only rational approach to iTunes is to assume it will either delete your music or otherwise render it unindexable, and prepare accordingly.

Except even NASA projects written to those standards had and have bugs still.

Apply Mission Critical/Life or Death practices to software can reduce the amount of bugs, but it doesn't take much searching to find medical software and other space software that has shipped with bugs, some with dire consequences.

I don't think the consumer software industry is prepared to take on those costs to still end up with bugs anyway.

Have you ever written software? Do you know what the acronym WFM stands for? Do I cringe every time I shrug sheepishly at the person in charge of customer service? My company's software is used by millions of people and man do they run into some of the most esoteric shit that we cannot produce in the lab. Once I got lucky and my sister's off-brand Android was displaying an edge-case bug we weren't even sure was real. Side-loaded a severely over-logging build and finally found the sucker.

Special iTunes build just means it dumps logs faster than... Never mind. The images that just popped into my mind weren't where I thought that was going as I typed it.

I am guessing WFM=Works For Me, in case anyone here is wondering.

It does, and I do cringe when after days of troubleshooting it's the best I can come up with. It's an exception, not the rule, but it happens.

Yup.

Also yup, customers will have the weirdest fucking edge cases on your software. Maybe not xkcd/1172 weird, but definitely out there.

We do this for customers all the time, customer has a special hardware/software setup, so we will send people out on site to diagnose odd hardware problems.

Exactly -- the broader the user base, the more likely it is you'll run into people with really bizarre and difficult-to-replicate issues. I've seen our main product grow from nothing to an industry standard, and darned if we aren't still seeing bugs crop up that had gone undetected for years.

People have been grumbling on blogs about this for some time -- google trends for "apple music deletes library" spike for about the last year or so. Wasn't it posted on HN?

Given Apple's original, incredulous response to Vellum (see links elsewhere in this thread), its 'known issue' status, the seemingly tepid response up-until-now, and the amount of time they have had to solve this most arrogant of bugs, my money is on the their appearance being some bright-eyed PR fix. I'd bet Apple's management doesn't give a shit about your files when they compete against Apple's pay service.

Edit: and good lord people! Empathize with me long enough to think about how, possibly -- just possibly -- someone might have a different interpretation! This is about tardiness and arrogance, not an insult to engineering. Sheesh

We get crash reports for our games on PS4 - there's one type of crash we got a few thousand reports for(out of a few million users, so statistically it's almost nothing). The callstack on that crash seems to be literally impossible, so we contacted Sony about it - they said it can only happen if the user's hard drive is damaged/corrupted. But there is no way for us to tell, so as far as the customer is concerned, it's our game that is crashing.

A similar problem occurs even when you control all the hardware.

I work for a company with tens of thousands of servers, and the number of strange things we run into is crazy. The exact same software running on the exact same hardware, and sometimes it just behaves differently.

We like to think that computers behave perfectly predictably, but so much randomness goes on.

Two threads, on one processor, out of dozens, out of hundreds of thousands of machines. Those two threads would do exp() in the wrong direction, somehow, but only with a certain byte pattern. A byte pattern that corresponded to some floating point number we used in a few places.

On that machine, everything using that library would stall. But only when it ran on that processor. Everywhere else, it was fine.

And yeah, it sucks. And it happens all the time.

Computers are hard.

I've had bugs relating to EMI issues crop up in testing before. The device in question had a compressor mounted inches from the touchscreen. It turned out the EMI from the compressor would generate phantom touch input on the display. It looked like a bug until I added a bunch of debug logging to the device.

Even if the computer is behaving predictably, the environment can do a wonderful job of making it go haywire.

Yeah, we sometimes forget that computers are physical machines that can be affected by their environment.

That's the key point. Apple should never be deleting stuff from users' devices. Never. Disk space is cheap. IP status is too ambiguous, and none of their damn business.

The deletion thing has been talked about for months, but no-one has been able to actually replicate it. I've never seen the problem, no-one I know personally has, and I'm sure if Apple could see it happening in-house they would be all over it.

Software is annoying.

I bet the guy turned on iCloud Music Library and hit replace.

That's one way to solve edge cases I suppose.

If your assassins tell you to watch Firefly they can finish their work before Season 2. It shows there are good people everywhere.

We came within moments of buying a customer a new phone in exchange for shipping us her old phone with a non-reproducible bug. Got lucky and found the issue just prior, so sent her a little something nice for the trouble.

Compared to the cost of extra developer time, this seems like a reasonable cost solution.

Oh I'm not knocking it for NR errors.

I fail to see how this is interesting

I'm confused by this article. The engineers both do not dispute what their rep said on the phone, and also admit that it was not user error. But they have not said that it should not have happened.

Maybe I'm misreading? Has anybody said that this was not meant to happen? They've accepted that it deleted their files, but was it a bug or not?

Apple sent two engineers out to look at something that couldn't be repro'd? Super unproductive.

Sounds like solely a PR move. A good one, I think, but a little strange. Basically just "let's listen to some tunes while we score these PR points."

The bugs that are the most difficult to solve are often unreproducible; and the hardest to find.

That's what makes them so dangerous - if it were easily reproducible, they could have simply taken a bug report and fixed the issue.

If paying customers are losing data and you can't reproduce locally, yeah, it's worth a few plane tickets.

These days it seems 99% of onsite visits for software problems are code for "We really care, and are doing are best". Mostly because of webex/etc desktop sharing software allow the support personnel basically the same access they get as being onsite minus the ability to move cables/etc around.

Some of the more enterprise software/hardware has "backdoor" functionality built in where the user presses the remote support button and it punches a hole through to the vendors DMZ where their developers can access the machine directly rather than having the user peering over their shoulder the whole time.

Nah, data loss is a serious bug and warrants investigation if the report sounds legit, which in this case it does. Why are you not patting Apple on the back for committing such resources to try to repro the bug? Why so cynical?

Reply on news.ycombinator.com