iOS 6.0 Causes CDN Overages | Hacker News Reader

iOS 6.0 Causes CDN Overages

labs.prx.org

157 pointswoodhull92 comments

I have never considered Apple to be a solid software company. They are a marketing company in the tech sector. If they made washing machines they'd be sexy as hell. In my years of exposure to Apple tech I have always been in awe of their industrial design, mechanical design and manufacturing prowess. Their software, however, often leaves much to be desired. I say this both as a developer and a user. From OSX to iOS there are amazing patches of software manure in plain sight. And it isn't getting any better.

addlepate wrote 37 minutes ago | link [dead]

OK. I bet you use a WebKit browser.

addlepate: you were hell-banned 100 days ago (unfairly in my mind).

Here's an easy clue: if your comments never get above or below 1 point for days, that means you are hell-banned.

The only thing you could infer from that is that you're not a bad enough poster to warrant downvotes.

Not really. It's a sure sign that you are hell-banned.

I mean, we can check statistically, and I bet we would find that nobody stays at 1 point forever in all his comments.

(And it's also upvotes you won't be getting).

I can't be the only one who upvotes good comments from hellbanned accounts in the hopes that the vote will have some impact.

Another clue is that nobody ever responds to you.

>I have never considered Apple to be a solid software company.

What does that even mean? That they make bad software? For one, they make the most popular certified UNIX OS in the world, and one of the two most popular mobile OSs. Their professional offerings are great too: Logic Pro, Aperture, Final Cut Pro, etc. In fact, FCPX aside, they are professional staples, with few competitors.

And huge number of desktop and the majority of mobile users use a browser they developed to the best game in town starting from humble open source origins (khtml -> webkit).

And that Clang thing, that FreeBSD recently adopted? Their work too, along with other LLVM infrastructure.

>Their software, however, often leaves much to be desired. I say this both as a developer and a user.

Care to mention any substantially better mobile OS than iOS?

(One might argue that Android is better. But substantially better, no way in hell).

Popularity means good software? Windows must be the best OS by very far then!

Regarding mobile OS, we all know this now depends on the number of "apps" available. Many may have made a superior OS, but adoption has been too low because of the lack of apps.

No, it doesn't. I think what you really meant popularity of a mobile OS depends on the number of apps available. In reality, a good OS really depends on the quality of its API. Code you can rely on to build great apps.

And, what do you mean by many have made superior OS? You meant Blackberry OS 10? Nah, it's vaporware. Or what you really meant WebOS? Nah, it's fartware. Nokiat attempts with weird funny names? Windows Mobile 8? Have you heard of DLL hell or whatever flavor of bad API Redmond throws out the windows these days?

We really have two mobile OS, Android and iOS. They both have good APIs, and also some dirty little bugs inherent of any new code base. And we have to deal with it.

>Popularity means good software? Windows must be the best OS by very far then! Yes, windows too, has had historically delivered the things its users cared about.

"Good software" is not some mystical status of code perfection, it's doing the things its users want it to do. So, yes, popularity equals good software.

Software, as any engineering effort, is measured by its utility and use. It's not like art where popular is not necessarily good. Unless you are some kind of dreadful "code poet", whose programs nobody uses.

But notice how I didn't just say that Apple's software is good because it's popular. For example, I challenged anyone to mention a "substantially better mobile OS than iOS".

>Regarding mobile OS, we all know this now depends on the number of "apps" available.

No, even without apps, the basic OS is just as important. iPhone, at it's introduction, had by far the best OS of every other smartphone out there, even without third party apps at all.

>Many may have made a superior OS, but adoption has been too low because of the lack of apps.

Really? Disregarding apps at all from the comparison, do you care to mention one "superior mobile OS", let alone "many"?

They bought Logic and most Logic fans will tell you that they've mostly just dumbed it down and dropped the price since then, turning it into "GarageBand Pro".

Not really much of a feather in their cap, particularly if you compare it to what the competition has been able to accomplish in the time timeframe.

>They bought Logic and most Logic fans will tell you that they've mostly just dumbed it down and dropped the price since then, turning it into "GarageBand Pro"

I'm a Logic Pro user (and my brother is a Logic Pro, err, pro). Logic today has nothing at all to do with Logic at the time Apple bought Emagic. It's extremely more polished and powerful, with far more capabilities (from amp modeling, to flex audio manipulation, to several new builtin plugins instruments, to 64 bit operation, to multicore enhancements, to OSC support). It's not even close.

And it's not like Emagic is out of the loop. Apple uses the same engineering team, based in Germany.

Anybody who says anything about "dumbed down" is more likely a dilettante relying too much on bloggers and pundits. Apple has only _added_ features to Logic and streamlined the interface. Nothing has been taken out (unlike, say, FCPX, where some features were cut due to the rewrite from scratch).

Logic Pro always comes with top scores when reviewed in trade mags (Keyboard, Electronic Musician, Future Music, Mix, etc), and if you read those, you'll notice the vast majority of professional musicians uses either that or Pro Tools. Cubase, Sonar etc doesn't even register, and Live is mainly used by electronica/live acts for it's particular strengths.

So, the "dumbing down" is just the BS meme that spreads around based on misunderstanding the FCPX situation, and from reaction to whenever Apple makes the changes to their pro UI look to match their latest UI schemes.

>Not really much of a feather in their cap, particularly if you compare it to what the competition has been able to accomplish in the time timeframe.

What exactly have they "been able to accomplish in the timeframe"? Logic Pro 9, 3+ years old, is still on the top DAWs around. Pro Tools is not even 64-bit yet, and just recently got support for native operation. And Cubase, is, well hardly touching Logic in plug-in power and workflow.

You're in the minority opinion here. I've been making music on computers for over ten years and actually code my own VST plugins so I'm pretty familiar with this market and Ableton, Cubase, ProTools, StudioOne etc have all left Logic in the dust with the possible exception of midi editing which was a core strength of Logic since the Emagic days. The engineering team in Germany seems to have mostly been assigned to GarageBand.

I know far, far more people that have dumped Logic for something else since the acquisition than the other way around. None of them can be dismissed as dilettantes. And I don't know anybody that uses it for serious audio work. ProTools continues to utterly dominate that market.

Aperture a professional staple? This is funny because I can see Adobe Lightroom installed everywhere but no aperture in sight. I'd say, the vast majority of pro photographers favor Lightroom.

>Aperture a professional staple? This is funny because I can see Adobe Lightroom installed everywhere but no aperture in sight. I'd say, the vast majority of pro photographers favor Lightroom.

I use Lightroom myself (part-time professional photograph). Lightroom just got better, faster, plus is cross-platform.

So, yes, Aperture plays second fiddle to Lightroom, but still Aperture remains the major competitor to Lightroom, and is used by many pros. Plus, it came out before Lightroom, and paved the way for this kind of photography workflow app as opposed to mere editing.

Not bad for a company that makes "bad software".

You don't say.

All that effort that went into humanizing BSD is absolutely laughable.

This is where Apple's closed nature is very annoying. Assuming the article is correct, it was bug introduced in 6.0, and fixed in 6.0.1. So Apple knew about it, investigated it and published a fix.

Yet they didn't tell anyone about it. Or the crippling effect this bug could have on their data allowances. Why not? Are they really that terrified of admitting to making a mistake?

This same behavior has been happening to me since before iOS 6 came out. I can show the same "206 storm" happening as far back as August of last year with iOS 4.3.5.

I used to work at http://castfire.com/, and we've seen that similar issue from a LOT of iOS clients. Stats become increasingly difficult, because we saw one unique request bounce with upwards of 50+ 206s for similar / overlapping media... and then it bounced through a cell provider, to a WiFi network, and back, etc.

(we identified by redirecting all requests through a complex process of assigning an ID to them)

Hoping this is fixed. Podcast metrics are tricky, and this has been exacerbating the issue.

The playback stuff definitely makes use of many range requests, but what we found was that they were being used in a way that vaguely made sense in other versions (i.e. downloading a few hundred kb to continue buffering, pausing the download, and picking up again). With 6.0.0, we're seeing actual storms of these requests happening many times per second.

To play devil's advocate, why would it matter? Is there any fix app developers could implement, short of blocking requests from iOS 6 users?

It would at least communicate to users that the upgrade to 6.0.1 is very important if they use the Podcasts app. Many may have put off installing the update because they thought their phone was working fine. A bandwidth eating bug has to be along the least visible.. in the short term.

This is the reason Forstall has left the building, iOS 6 is a disaster:

    * Maps was not ready for prime time (so bad it required a public apology)
    * Podcast was not ready for prime time.
    * Passbook was not ready for prime time.
    * App store is a step backwards.
    * Battery and WIFI issues.
    * Soooo many bugs.

There are usually problems but this is the first time I've seen so much utter crap shipped including tons of minor bugs and issues (well Mobile Me was pretty bad :-)

I'm using Audible, Downcast and the built in Music app where I used to just use the Music app. iOS 6 was a huge step backwards for me.

Rather than waiting until it's ready (think copy/paste on iOS) Forstall was responsible for shipping a mess. Hopefully this will fix itself with new leadership.

How do you know that Forstall was the person advocating a yearly OS upgrade cycle? It seems just as likely to me that this rigid schedule was forced onto him and he couldn't deliver. Things like OS X Lion (a huge change under the hood) and the Maps places database just aren't things that can be built to a deadline.

Well, a mess was shipped and he was the guy in charge. So I'm assuming he was the guy responsible. If a mess really was forced upon him he should have resigned instead of sticking around to ship the mess.

I have in that exact same situation, "This will not be associated with me, I'm out"

Now you are talking about saving face, not about saving iOS. Maybe he was the best guy for the job and he still couldn't do it.

Is it just me or are apple making more dumb mistakes lately? There's this and that ajax caching issue from a month or two ago.

Perhaps I'm just having selective memory?

I've had the exact same feeling. It seems like their quality control has gotten worse over the last year or so.

I think Apple has always had the tendency to make dumb mistakes, but I will say that I think reporting on it has changed. Remember all of the jokes about "It's not a bug, it's a feature" back when Jobs was there? Apple is by no means immune from mistakes and I don't think they ever were.

Exactly. The reality distortion field has collapsed and everyone is less willing to forgive mistakes. I mean come on, remember the MobileMe launch failure?

That said, the famous Apple attention to detail does seem to be slipping, I sincerely hope this is just a transitional period and they tighten their game back up.

People also seem to forget how much of Apple's history was spent as a nearly faling tech company. Each little mistake is being reported to the point of saturation and while I think they're worth mentioning, every article seems to be a "beginning of the end" piece. As companies expand and begin to offer more and more products and services, it makes sense that their amount of bugs and mistakes would grow. When looking at gross amount of bugs and mistakes, you have to look at growth in offerings before major concern.

I dunno, how crummy the App Store has become is a perfect example of this. The App Store is one prominent way they make loads of money and build loyalty to the ecosystem, and yet it is a shitshow on IOS6...the responsiveness is as bad as I've seen on non-iOS software, but for me, the two really inexplicable failures are:

1. the use of modals to show individual apps on the iPad. So if you get to an app page and accidentally touch outside the modal, the app page disappears and you have to go back to Safari to find the link that brought you there.

2. On my brand new iPod, the Genius View has never NOt crashed. I know Genius (and most of Apple's search services) are poor to begin with, but it's a prominent button on the nav. How does it just not work on a brand new device?

3. I'm also guessing the lack of a light sensor on the new iPod was the result of a screwup in the dev cycle...I would've never guessed that its absence would be such a deficiency but when you can't see a single thing on the iPod merely because you stepped from indoors to outdoors...to the point where you have to go back inside to adjust the brightness...that's just infuriating.

I can't remember the last time I've been infuriated by an Apple device...I feel infuriated all the time with my expensive Sony camera and how its video button is right where you can accidentally bump into it when shooting a pic...which prevents you from shooting a photo. The last few Apple devices I've had (two laptops, 2 phones, 2 iPads, countless iPods) have always been satisfactory in terms of polish...not iOS6 nor the new iPod touch

> the use of modals to show individual apps on the iPad

This bugged me a lot after iOS6 update. Also the small things such as when there are App Updates and badge shows up on the App Store icon, clicking it takes me to Featured Apps section - not updates as one would expect. On Android for instance, the update notification takes you directly to a screen allowing you to update as expected.

The thing that continually annoys, and occasionally infuriates me, is the f'ing no-button trackpad on the MBP: a dozen times a day my finger or thumb slips on the thing, just as I'm clicking, and it misses, or hits the wrong link.

Disable the button altogether, and turn on tap-to-click. I used to hate it, but two-finger right-click works well, and the new 3-fingers-to-drag gesture means I never have issues with dragging, which was the main pain with tap-to-click.

Your Sony: is it an NEX-7? There's a firmware update just out which fixes the video button problem: http://blog.sony.com/alphafirmwareupdates

You have made my day...that blasted button is so clearly in the wrong spot that every time I bump it I secretly hope the Sony engineer responsible has been fired

Ever since I started using Apple gear, I've found apple's software quality is way below hardware. iTunes on Windows was always a disappointment.

>Is it just me or are apple making more dumb mistakes lately? There's this and that ajax caching issue from a month or two ago. Perhaps I'm just having selective memory?

No, it's not just you. It's the whole tech pop culture sector that has selective memory.

Apple did as much mistakes on the old times. It's just that it's not the underdog anymore, and all the hip kids are looking to fry Apple for those, instead of understanding that, heck, mistakes happen in every software/hardware offering.

The iPhone 4 antenna issue? Not even a blip compared to the iBook G3 logic board fiasco. But everybody forgets those things and adds it to some all-encompassing theory of decline...

Same with the notion of the "non-innovative update" for iPhone 5, etc.

The forgotten that less than a decade ago Apple did even far more minor incremental updates to the iPod line (a new click wheel, some more storage, a little thinner, now with a color screen, now it plays video, etc) and everybody was overwhelmed everytime a new device came out.

Suddenly they expect miracles out of every Keynote.

Author of the post here, happy to answer any questions. I'm pretty confident in our findings here, and some of the behavior is just bizarre.

Does 6.0.1 fix it?

The article says that the problem appears to be fixed in 6.0.1, at least for wifi.

We have some ideas on how to test 3G with 6.0.1, but haven't proven it one way or another yet.

From the fine article: "We have been unable to reproduce the issue using iOS 5 or using iOS 6.0.1".

So the problem's already been fixed as far as they can tell.

True. The problem is that we are seeing > 20% of users still on iOS 6.0.0, and still chewing up bandwidth. Upgrade, please upgrade.

I saw similar behavior when I was working on the backend for an IOS app.

We were able to track it down to a retry mechanism for streaming content when the server was responding too slowly. Maybe you are running into something similar.

I'm seeing this line in my log as well, this person downloaded about 750MB and they don't seem to be on iOS 6.0.

  Reqs  User Agent
  5,378 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.14 (KHTML, like Gecko) Version/6.0.1 Safari/536.26.14

Hmm, so this explained my 3G data overage last month. I have 6GB/month of data, and usually I'd be using around only 700MB. But last month my iPhone 4S (running iOS 6.0) consumed nearly 8GB in total, resulted in quite some overage fee.

Should I ask Apple for a refund for this? :|

Same problem here. My monthly data usage increased by 10x after I upgraded to iOS 6.0. It's annoying that there's (apparently?) no mention of this issue in the 6.0.1 release notes.

Maybe Apple was trying to stimulate the economy.

Try your data carrier, rather than Apple.

I had a similar experience. I normally consume 750-800 GB a month and on my first full month of iOS 6, Verizon said I consumed 8.89 GB. Luckily i'm grandfathered into an unlimited plan.

Last month I moved and didn't have an ISP at home for the first 3 weeks or so. I set up tethering on my phone and a 5GB data plan. Yesterday I got a notice that I was over my data limit. I haven't done a whole lot of web surfing in the new house, but I have listened to a ton of podcasts using the Podcast app which I have never done before.

So if I just blew through 5 GB of data in 2ish weeks and the only thing different was using the podcasts app...

(I did watch 3 episodes of walking dead on Netflix which obviously contributed to the data use, but a 45 minute episode can't be that big)

Netflix's 720p streams use 2.3GB/hour, which would certainly put you over 5GB. I'm not sure about their lower quality streams though.

Hmmm, maybe that is it then. I'd be surprised if I was getting that data rate on my 3G connection (iphone 4s, at&t).

I'm still shocked it happened so quickly though. I guess 5 GB isn't what it used to be! ;)

Interesting. I was using the Podcast app too, which I believed was the reason for the data overage. It shouldn't be, though. A few podcasts are definitely under 1GB. This AV Foundation Framework issue seemed to be the root cause.

Same here, noticed a huge spike in bandwidth for my startup's shows hosted on CloudFront. 206 responses are flooding my logs and I moved everything off of CloudFront for my users to triage the cost. This breakdown explains it all: http://cl.ly/image/2H3u3u2O333g

Talk about a pain, not to mention the 10x bill from Amazon that came as a surprise.

Exactly - we see this causing a crazy number of 206 calls.

"I moved everything off of CloudFront for my users to triage the cost."

Can you explain more? Was there some cloudfront behavior exacerbating the issue? AFAIK this is just a client (mis)using range requests and CloudFront replying as requested.

You are correct and I should have wrote that note differently. I moved some high traffic files to some endpoints I have where bandwidth isn't a concern to change the formula a bit (primary) and the rest are back to using basic S3 for now. The bandwidth is still being consumed, however now it's easier on the wallet.

After looking more closely at the differences for pricing between S3/CloudFront I realize it's appears cheaper to be on CloudFront. I assumed wrong that CloudFront data-out was billed in addition to S3 because of the way it's notated on the usage page ("AWS Data Transfer (excluding Amazon CloudFront)"). To prove it I ran two reports on CF (left) and S3 (right): http://cl.ly/image/0q0y3u2X0g1a and you can tell where I made the change, as well data is only counted on the service in question and not on both.

Can anyone else confirm the above?

Yeah, that's a bit confusing. Using Cloudfront with an S3 bucket is double billed, the first time. On a cache miss Cloudfront pulls the object from S3 and serves it to the client. S3 bills the regular data transfer out. Cloudfront then bills the regular data out to client rate.

On subsequent requests, cloudfront cache hits, you're only billed by Cloudfront. Cloudfront request + byte rates are cheaper than S3 in Us-east-1, IIRC. So on popular or high ttl objects it's cheaper to serve through Cloudfront. On low ttl or low rps, like a few requests per day, it's cheaper to use standalone s3.

The same origin + CDN vs CDN Hit math applies to EC2 as well. I do wish the billing was clearer in these scenarios.

Thank you so much for breaking that down.

Our very small outfit at verbmill.net is seeing similar signs. For instance, 700MB+ downloaded by a single client when an individual podcast is a maximum of ~50MB. Checking my logs now to see if I can narrow down device/version number. All of our files are hosted on S3.

Ominous example.

  Reqs	Total	Avg	B/s	Client
  666	2G	3M	127K	mobile-198-xxx-xxx-088.mycingular.net.
  666	AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)

I hope this user is on an unlimited plan :(

Wow, that is truly unfortunate. How long did it take for that 2GB to be chewed up?

I believe this is directly related to phone data usage overages as well. Having used an iPhone 4 on ATT for 2 years, I could almost guess my data usage perfectly each month.

By the 2nd day of using my new Verizon iP5, I was receiving warning texts from Verizon that I was about to exceed my data limits.

Thinking it was me, I increased my plan. That only stopped the bleeding for another few days.

Oh, and I installed the "fix" that was sent out to supposedly repair the known issue. (I was always on Wi-Fi networks but still being charged for data.)

I recently called Verizon to dispute the data. They quickly erased all of my data overages. Snap, like that. The man I spoke with called a technician on my behalf.

The people who can erase overages do not officially have this issue in their database as something to be aware of. But the technician he spoke with said there was absolutely a problem, and both Verizon and Apple knew about it and were working on it.

Then, I was given a "diamond ticket" to absolve me of any future overages until a fix is found.

Makes you think twice about those graphics where iOS devices take a big piece of the pie of the "internet usage"

No it doesn't. Those aren't tracking bytes streaming over the internet, but rather web browsers visiting pages.

Yes it does.

Even ignoring the bytes, it looks like one client may show up in the web logs as downloading the file a dozen times. If you aren't careful about cleaning up the stats, that may overrepresent iOS.

I don't have anything wildly new to add, but I'm part of a podcast hosting site (http://Castmate.fm) and we've seen this bigtime, it's crazy.

We host some high-traffic podcasts with a huge range of listeners, and I've gone through the logs and regularily see stuff like NUMEROUS listeners making 4000+ http requests in 30 minutes, all the 206 calls everyone here is reporting, it's just nuts, so bizarre.

I feel terrible for all the people getting hit with overages, it's so gross. I did some gorilla math the other day and feel pretty confident that this has boosted our network usage by ~25%, I should have narrowed it down to just the affected iOS 6.0 devices, must be astronomical.

This just seems to be how they handle audio media. It's not even anything specific to podcasts. My own scanner stuff is just a web page, and it gets hit by this from time to time. Each call is a single file, it's static, and there's nothing funny going on with it. It looks like the playback stuff is jumping around in some attempt to figure out exactly what it is first. It seems like it would be far better if they always downloaded at least some reasonable quantity and then decided to bail out later if it seemed overly huge, but this is where we are.

To me, it feels like someone who's trying to be fancy and do things like seek() in a stream. Just because HTTP has progressed to the point where you can effectively do that with Range: requests doesn't mean it's a good idea. Offering seek() type behavior for a HTTP resource seems like a great way to get people to use it and not even realize what sort of messes they may be creating. Leaky abstractions, eh?

Here's just one example from last week. Note: the file is 7001 bytes. There's no reason for this insanity, and yet here it is.

    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"

Yep, as I mentioned in the article this is the AV Foundation framework - presumably used in all Apple apps since it's the official API they recommend. Doesn't surprise me that the browser would be affected.

The weird thing is that, when we first started doing the tests, it would happen immediately on pressing play. After some period of time running the tests, it started happening later and later in the process, indicating - what? No clue.

It's not just audio files; we've been seeing the same pattern (though actually even worse, as they request not only overlapping ranges, but the complete file, multiple times) on iOS5 for MP4 files: http://stackoverflow.com/questions/6094556/mobile-safari-mak...

This is exactly what I saw with the Podcast app last week. Used up 2GB in about 6 hours and put me into overage. I bet the class action lawyers are already circling.

I am anxiously awaiting an update to the Podcast app which fixes the issues that I am having, but perhaps I'm too optimistic. I listen to some podcasts from the BBC and the app doesn't recognise that a new daily show with the same name is new, the only way around it I have found is to delete my subscription and manually find the podcast every morning. Apart from that the thing crashes and runs very slowly at times.

Now that explains the sudden spurt in my 3G data usage in the last 3 weeks. This forced me to switch off my 3G :(. Hope Apple releases a fix soon

We (PRX) haven't seen any evidence to show that it is still an issue in iOS 6.0.1, so you should be all set if you've upgraded, or an upgrade should fix it. If you are upgrade already and seeing overuse, please comment.

I was (and still) using iOS 6.0.1 when I started seeing the surge in data usage

Hmmm, I had to turn off the LTE on my iPad two days ago because I hit my limit 10 days prior to my monthly renewal. I upgraded to 6.0.1 last week, but I wonder if this bug explains how I managed to chew through my data plan faster than ever before.

The iOS 6.0 podcast bug apparently caused huge overages for GiantBomb as well. They're discussing it right now on one of their live streams; it apparently was so bad that their corporate/finance people are upset with them.

NoAgendaShow.com has been discussing this since iOS 6 launched. Don't upgrade.

As pointed out above, 6.0.1 fixes this, so upgrading shouldn't be an issue.

Good point. But then you have Apple Maps.

Ouch, elsewhere on the internet it seems like people are actually getting hit by this. Hopefully damage won't be too bad. Good research, PRX!

Wonder if the CDNs don't have any safeguards in place to avoid single client downloading same resource multiple times?

That doesn't seem practical. How would they know if it's the same client? Just watching for multiple requests from the same IP will result in a lot false positives (e.g. most college students will be SOL). And what would they do if they could detect it but the client didn't realize it already had the file? Just drop the request?

Load balancers can insert cookies to identify clients uniquely. (That solves the proxy/single-ip-multiple-clients problem - multiple clients from one proxy IP can still be uniquely tracked this way.) The newer load balancers (F5/iRules for e.g.) are also capable of doing per resource, per client throttling using the inserted cookies or session table.

Server side state per client is insane. Pushing it client side would be what, a kb? Wih an average object size of 50kb that's 2%. I doubt CDN customers would be willing to take a 2% cost hit for this feature.

Now, assume we solve all that. The state is pushed on the client. The performance hit is so small as to be free. And the CDN eats the dev and opex. So what does the CDN do when "bad" range requests are detected? Throw a 400? What customer is willing to break all IOS 6 clients?

Also, iRules. Serious LOLs.

For starters, average podcast download is 40Mb or so. More than 50Kb in any case. Next maintaining state per client isn't anything new - it might not be ideal for CDNs but many LBs keep state for things such as maintaining session persistence etc.

If there is a potential for clients intentionally or unintentionally jacking up your CDN bill - I am sure somebody has a solution to prevent it - especially since money is at stake.

You laughed at iRules - I got that part, but what about iRules doing HTTP Request throttling? Far as I know it can do that based on source ip and port, uri etc. which should work for this kind of scenario with some modifications.

Your average podcast source material may be 40mb. Clients will most assuredly not be fetching that all in single request. Your state transfer overhead is per request. The 50kb estimate is an educated guess at the average size of an http response from a CDN. You have to keep state on every request, otherwise how can you detect your "bad request" patterns. Now, you could only stick your state on to Range'd requests. But a lot (most?) of the customers affected by this are doing a significant portion of requests as ranges already.

IRules, specifically, don't work for any reasonable packet rate. Every request must now come off the nic, across the bus, hit CPU, hit a few times memory, and back. Tracking state, in general, kills packet rates. There's no guarantee that your flow is going to be hitting the same interface, interrupt, processor, or even host. At every one of those levels shared state dramatically increases complexity and reduces your max possible packet rate. Silicon really can't flip bits that quickly.

Which gets us back to the business question. So you've found a "bad" client. What aceptable action can you take? Throttle all iOS 6 users? Throw 400s?

To be honest here this excess cost is going to be absorbed in three places. 1) end users will suck it up in data charges because they have no alternative 2) sites will eat the bandwidth charges. They can't passit on if they have no directly associated revenue. Or they don't want to lose customers. 3) CDN/providers will take a relatively small hit issuing credits to keep their customers happy.

Notice who won't lose a cent here? Apple and other broken client providers.

Reply on news.ycombinator.com