Apple Q&A on gathering and use of location data (Apple Press Release)
apple.com
apple.com
It's going to be unfortunate when I can't do this anymore because of people blowing this issue out of proportion. I hope Apple will at least provide the option of caching this data for longer than 7 days.
That file on the iphone is just a local copy of the relevant parts of the central database.
Sure. It might be 10 seconds instead of one second now, but that's a reasonable tradeoff compared to the huge log it currently makes. IMHO.
I don't think it's a privacy concern to store such data locally on the phone. In any case, it should be resolved by letting the user decide how long to cache the data; that would make everyone happy.
However, it is included in the iTunes backups. Those can be read -- but only on your computer, not on the iPhone itself.
(An exception to the above would be a jailbroken phone: the packages installed via Cydia would not have the same restrictions, although the recent jailbreaks do not remove the restrictions on apps from the App Store.)
The solution is to properly protect your data on your phone and your computer.
Any malicious desktop tool can easily find the location cache in unencrypted backups. Modern Police Forensics tools (http://www.cellebrite.com/) can easily extract non-encrypted data from phones in minutes (see Michigan Police).
That Apple stored this growing set of user-data in cleartext on the device was as stupid as Sony storing their customer's personal information in cleartext (or weakly hashed) on their servers.
Either bit-recycle the information that's not immediately relevant, or strongly encrypt/sanitize it. This shit isn't rocket-science, folks. Otherwise it's a liability and potential PR nightmare in the making.
We're now still in the "wild west" of personal data records. Once these issues start to snowball and real-life consequences happen, people will clamor for litigation, which given politicians will be over-reaching and ham-fisted.
Corporations with hundreds of millions of users' personal data should stay in front of these issues unless they want to wade in a regulatory mess (see Google's mis-steps with wifi packet data).
You also say that you have little way to protect your data, and then in the next sentence tell me how to do it.
Are you really trying to evaluate the situation, or are you more interested in attempting to criticize in any way that you can stretch words?
How is that secured?
I don't know if this is because the iPhone's GPS functionality is crippled, or if it's a software limitation. I'd imagine that if you wanted to overcome this limitation, you'd need to jailbreak your phone and use 3rd party mapping software at a minimum.
*Note: once it has located you, it will generally track you correctly while you're moving, even if you leave the area where you had internet connectivity.
The almanac is valid for 180 days, so you only need to wait 12 minutes once. From then on, you only need the ephemeris from 4 visible satellites, which takes less than a minute to download.
It confuses me that the iPhone wouldn't be able to do this, because I can buy a $10 microchip that's the size of a dime that can do it. Give it electricity, wait 15 minutes, the location comes out the other side. I would be shocked if the iPhone were incapable of doing this.
How do you know when the 15 minutes starts, though? Dedicated GPSs tell you things like how many satellites they have in view and how accurate their fix is, but dumbed-down devices like to pretend location is magic. I'd hope they'd at least provide the gritty details through CoreLocation so a user can buy an app if she wants to know which bench outside the train station to sit at sipping coffee while the phone orients itself.
The best you can do with CoreLocation is set desiredAccuracy to kCLLocationAccuracyBest and wait for it to fall below 100 feet with no way to find out if it ever will.
With the iPhone I get a decent location in seconds and a more accurate one in ten. Indoors.
The iPhone is not logging your location. Rather, it’s maintaining a database of Wi-Fi hotspots and cell towers around your current location, some of which may be located more than one hundred miles away from your iPhone, to help your iPhone rapidly and accurately calculate its location when requested
This actually makes sense - looking at the logs on my iPhone and iPad, the locations where pretty far away from the places where I usually roam, and actually there was a very great deal of places where I've never been, not even close.
How does that make sense? It should only cache the locations of cell phone towers it was in the range of. Are you seeing data that was more than 100 miles away from the places you were at?
Yes, actually. I used the app that was posted and, even after I modified the code to stop fuzzy-ing the data, I was thinking to myself "well, I've never been anywhere close to there".
It is nothing but a very neat cache. The nerd in me is sad to see it go.
Apple hasn't explained the algorithm they use to determine what location data to send to the phone. Perhaps they're saying "100 miles" because that's what they do -- send tower data for up to 100 miles around the current tower, depending on tower density for that location.
Reducing the size of the wifi location cache to a mere 7 days could severely reduce the usefulness of that feature.
I really hope they're not killing a great feature because of some hysteria and bad reporting.
If it's going to harm use, I don't doubt Apple would turn that 7 into a 14 or a 30 without batting an eye.
I have seen this more and more lately, the standard reaction to anything Apple in the tech community is pitchforks. I have a feeling subconsciously we all want Apple to fail at something and try to latch on to anything remotely blamable.
I think that's a counter reaction to a whole bunch of people who think Apple can do no wrong at all.
This press release doesn't change any of those points, it merely places the blame on bugs (which wouldn't have been patched had the story not been uncovered).
Isn't this comparable to the Google/Buzz fiasco, which was met with similar (deserved) scorn?
That's a pretty cynical reaction. Why would you make that assumption? Do you think that if someone had alerted Apple privately to the bug they never would have gotten around to fixing it?
But that's an incredibly negative view isn't it? By imitating Apple, the other manufacturers simply aren't contributing anything meaningful. It seems unlikely that Apple's failure would make them suddenly unleash a wealth of creativity and contribution on the world. If they had it, why wouldn't they be using it right now?
Wouldn't it be better for Apple to succeed, and the imitators fail? That way, we'd get the best of both worlds. Advancement of technology from Apple, and a clear marketplace signal that imitators are not wanted.
New companies and investors would who wanted to succeed would no longer be tempted to imitate but might actually innovate.
No. This data is sent to Apple in an anonymous and encrypted form. Apple cannot identify the source of this data
I hate it when companies use 'encrypted' like it would somehow make your data more secure in their hands. They mean here they use something like SSL so snooping the traffic is impossible, but of course they can still read what you're sending, otherwise the information would be useless to them.
The claim that apple can't identify the source of the data is also highly dubious. If they wanted to, they could probably correlate your IP with the IP used to access your iTunes account. That they don't do this is one thing, but claiming that they can't is something else.
EDIT: come on HN, since when did we start downvoting stuff you don't want to hear? This is a valid point, if you have critizism just leave a comment. If you want to downvote something, do it on the summary comments. Sheesh.
In order to populate their wi-fi/location database they really don't have to send over a person's IP to the server. In fact its something they don't want I would assume as it adds absolutely no value to the data. They already know who you are and where you are don't they?
They have access to it sure. But they have access to it through a lot of other methods. In fact they even have your credit card information. You gave to it them :)
This is not about ability. This is about trust. They say the data is anonymous which means they are obligated to disregard the IP address and to not log it anywhere.
Sure it is. That's exactly what the following statement means: "Apple cannot identify the source of this data". How can you interpret that as anything else but their ability of identifying the source?
The statement is obviously not true; if they were forced to, by a government agency for example, to track the location information from a user from that point on they COULD; saying they can't is wrong IMHO.
The correct thing to say here is that they can, but they don't, unless forced to. But I guess that isn't the message Apple wants to communicate.
You must be new to HN, any post that can be construed as criticizing Apple, whether legitimate or not will either be ignored or downvoted. On the other hand, anything justifying Apple's actions, whether true or not will be heavily upvoted.
What will we all hate on Apple about next?
That said, I cannot imagine that if they'd contacted Apple about the issue they would have gotten a useful answer without the publicity.
Bottom line though is that any sufficiently sensational story gets traffic regardless of its truth or lack of attempt to even verify the veracity of the alleged problem.
"The big question, of course, is why Apple is storing this information. I don’t have a definitive answer, but the best at least somewhat-informed theory I’ve heard is that consolidated.db acts as a cache for location data, and that historical data should be getting culled but isn’t, either due to a bug or, more likely, an oversight. I.e. someone wrote the code to cache location data but never wrote code to cull non-recent entries from the cache, so that a database that’s meant to serve as a cache of your recent location data is instead a persistent log of your location history. I’d wager this gets fixed in the next iOS update.”
http://daringfireball.net/linked/2011/04/21/andy-ihnatko-loc...
"The key question for Apple: Given that this file was widely known among iOS forensics experts back in September, why does it still contain historical (as opposed to just recent) location history today?”
http://daringfireball.net/linked/2011/04/21/watts-martin
"Android phones store the same type of location information, but, unlike iOS, Android’s cache only contains recent entries — which is to say Android is doing it right.”
http://daringfireball.net/linked/2011/04/21/android-location...
"Really good questions” [About Senator Al Frankens letter to Steve Jobs on iPhone location tracking]
It's certainly reasonable to question their motives.
Can Apple locate me based on my geo-tagged Wi-Fi hotspot and cell tower data? No. This data is sent to Apple in an anonymous and encrypted form. Apple cannot identify the source of this data.
I think a lot of folks who spent money on Apple products are going to be happy with this, and for that I'm glad.
But I didn't find this release adequate. Apple is not tracking me -- they are keeping a time-stamped list of nearby access points on my device, which effectively is a huge breadcrumb trail of everywhere I've been and when. Apple doesn't know it's me -- because the data is encrypted, which makes no sense at all. Whether data is encrypted or not is meaningless. Can I go to the Apple server logs and track incoming downloads and associate them with the data or not? I strongly suspect the answer is "yes". If not, that's great, but that wasn't described here.
The killer omission? That Apple has been doing all of this -- which is at the very least controversial -- without informing the users in a manner in which they clearly understood it. The response we see is simply a reaction. The "bug" here is getting caught.
I don't necessarily see anything nefarious at work, but I'm troubled with the idea that Apple was keeping a list of my whereabouts (the nearest access point, for those of you who are literally-minded) without my knowing it. That's a pretty serious breach of user trust, no matter how many times it was covered in the 47-page lawyered-up doc that nobody reads.
But like I said, folks are willing to cut Apple lots of slack, and they deserve it. But hell if I'd want to see something like this happen again, from any manufacturer. I'm not so sure that vendors are getting the point.
Then, I hope that nobody got convicted on the basis of such "accurate" informations...
Yup! The crowdsourcing part (how they initially assembled and now maintain the hotspot database) needs a bit more clarifying. The release glosses over it and people everywhere are instead fretting about the file, even though it's just a cache, a mere fragment of Apple's data, not theirs.
The iPhone UX makes it very obvious that apps are using location services but it never conveys that the iPhone user is contributing data back to Apple, even if anonymously. This is new information for everyone, I guess.
Furthermore Apple implies that they uncovered the but themselves.
The reason the iPhone stores so much data is a bug we
uncovered and plan to fix shortly (see Software Update
section below). We don’t think the iPhone needs to store
more than seven days of this data.
It's a very interesting comparison as well with Google who (for example) with Buzz and the Wifi sniffing incident did complete and total mea culpas without reservation. It didn't particularly seem to help Google in those cases so it's hard to argue that Apple needs to do the same, but I do feel they would be better off with a simple and direct "sorry" and a promise to do better rather than a "oh it was just a bug, and we found it ourselves anyway, and by some definitions you're all wrong anyway so there!".
Actually it is because the data is made anonymous not because it is encrypted. If they do this correctly they really don't know it is you.
Read again. They are not. Its not timestamped when "you" where there, but when some anonynous iPhone picked up that particular hotspot. It got then uploaded to Apple and subsequently downloaded onto your iPhone, so that your iPhone can find its location easier.
Wasn't that kinda the obvious reason in the first place?
The point is that the database on your phone is a subset of the greater (global) database mapping wifi and cell access points to location. The local database contains timestamps of when your phone downloaded the information. The global database only contains a list of access points and approximate locations of those access points.
That way, when your phone sees an access point it can look up in the database (locally or globally) and see if this access point has been pinpointed, if it has it will be another tool for your device to provide you with a proper location.
In other words: animating the points in your database will show the first time your phone downloaded information about a specific access point, which will give you a trail of your movements (to a certain extent limited by the factors mentioned above).
Your email is on your phone. Does this mean that Apple has your email? Clearly not, or perhaps we should have 'emailgate' and prevent phones from downloading emails more than 7 days old.
Apple was never keeping a list of your whereabouts.
- reduces the size of the crowd-sourced Wi-Fi hotspot and cell tower database cached on the iPhone,
- ceases backing up this cache, and
- deletes this cache entirely when Location Services is turned off.
Looks reasonable to me. The only thing missing that I'd like to see is an option to opt out of the tracking data (anonymous or not).
(I wish Apple would take that same stance with regard to crash logs. If you want stuff from the App Store, you have to let iTunes send your crash logs to the developers, without having to ask for permission first. There really is no good reason why anyone would need to opt-out of that.)
No I don't have to.
I am currently working on an offline positioning system based on a (hopefully complete) dataset of all GSM-Cells in the world (http://myapp.fr/cellsIdData/). The sqlite3-database is currently 25MB big with some room for optimization. I do this for Android but the iPhone also has enough space to hold this data. Although I think it would be impossible with iOS because of missing APIs to be notified when the cell tower changes.
I'm sure that cell towers are enough for Assisted GPS to get a quick fix.
If people would have found a data set of all cell towers ids and coordinates no one would have had concerns about privacy.
Tracking data is only anonymous if it cannot be de-anonymized. Since I cannot check that this is not the case (with Apple, Google and Skyhook) I have doubts about the so called anonymity.
edit:
I am looking for a comprehensive dataset of wifi access points MAC-adresses and coordinates
Whether or not the collected data really makes the positioning that much better, I don't know, but Apple apparently seems to think it does.
Given their description of how the cell tower/hotspot data is used, it seems like deleting "this cache entirely when Location Services is turned off" is the opt-out you are looking for.
I wish people would stop saying this. Of course it's tracking data. There's nothing wrong with that. One of the phone's features is tracking your location so you don't get lost.
Personally I think Apple has overreacted. They just need to provide a configuration setting for the size of the cache, and more importantly not include it in backups. A cache is not important enough to back up. But as far as the size of the cache, the defaults sound sane enough.
Hell, I get a text message from Rogers the second I land in the US to let me know that roaming rates apply. The fact that I have a device that has to talk to another end point means that I can be tracked.
Apple is now collecting anonymous traffic data to build a crowd-sourced traffic database with the goal of providing iPhone users an improved traffic service in the next couple of years.
"Tracking" vs "traffic". [shrug]
But I'd still strongly prefer to be able to opt out. GPS still works to find location without this data, it's just slower. I should be allowed to take that hit, for the trade-off of not sending in data. As biafra says below:
Tracking data is only anonymous if it cannot be de-anonymized. Since I cannot check that this is not the case (with Apple, Google and Skyhook) I have doubts about the so called anonymity.
"Jobs: If people don’t want to participate in things, they will be able to turn location services off. Once we get a bug that we found fixed, their phone will not be collecting or contributing any crowdsourced information. But nor will it be calculating location."
So it looks like you will be able to opt out completely, if you really want that (but then what's the point of even having an iPhone?).
Other than that, I think this is a good rational response to the situation.
2. Speculatively, the way this seems to work is, that the phone identifies a tower, say, with ID12345. It then looks up the crowdsourced database for the tower with this ID, and queries it for all towers/hotspots within X miles radius. The result of the query is logged into the consolidated.db file, along with the current timestamp.
3. I don't know about the 100 miles number, but for me, in an urban setting, it certainly seems to be accurate upto approximately a mile or so, that together with the timestamp, gives a reasonably accurate picture of where I've been, and when.
As a devils advocate, if I'm a developer writing some sort of logging routine I'm including a timestamp without even thinking about how or why it would be used. Its a logger its supposed to have timestamps ;-)
This was one of the best responses out of a big corporation I've seen in ages. They even explain the function of it, not often that you see that.
Interesting. Apple generally does not pre-release information about upcoming products, at all. They must have felt their hand forced in this, or Jobs is not at the helm of this press release (which I'm sure he is).
The problem with traffic info right now is that it's always either laughably late (see sirius/xm traffic info) or it's based on eyes-in-the-sky style updates from radio, which you can only get every period of time.
I also like the idea of using millions of iPhones' locations to get comprehensive traffic data.
However it's vaguely reminiscent of the cell phone sonar surveillance system in The Dark Knight.
1. Why is Apple tracking the location of my iPhone? Apple is not tracking the location of your iPhone. Apple has never done so and has no plans to ever do so.
If you are "collecting anonymous traffic data" that means that you have to collect a phone's position and velocity, as well as time. That sounds like tracking to me.
Sure, it is probably anonymized and encrypted and aggregated into some probabilistic model, but it's still tracking the users' movements. I wouldn't be surprised if it's all stored somewhere server-side, that we only will find out about when something happens to Apple like happened to Sony's PSN network a few days ago.
http://www.fool.com/investing/general/2010/08/04/apple-drops...
"Apple apparently has its own Wi-Fi location information, presumably culled from the daily movements of iPhones around the country, that it thinks is good enough for its own devices."
That sounds exactly like what is described in the press release.
I mean, really? That's just argumentative. My mom is gonna look at the location history visualizers people wrote and respond "Really, Apple? Cause this looks very much like log of my locations".
Somebody at Apple's PR needs an ass-kicking. This ought to be a video with a short transcription from someone on the phone team (not Jobs) that just explains it without getting defensive of semantically tricky.
A bug _you_ uncovered?
;)
https://alexlevinson.wordpress.com/2011/04/21/3-major-issues...
• The internet claims Apple is tracking all iPhone owners!
• Steve Jobs interrupts family time to exclaim "WTF!"
• SJ: "Minion! Verify this claim!"
• Minion: "It is true, to the extent that we keep large, possibly unbounded, volumes of cell tower and wifi access point in the cache for the purpose of…" SNICKER-SNACK… thump.
• SJ: "Engineering, fix this. Marketing, communicate this.", returns to family time.
• Engineering to engineer: "Fix this."
• Engineer to self: "I do not know why this happens, I will search and uncover this bug." <<<--- there, that is where the bug is uncovered. He fixes the purge code from whatever simple or broken strategy the first coder used, perhaps deleting the comment that says "// might need to prune the cache, but the OS probably does that when it gets too big"
• Marketing: "What? We were busy hiding the links on the home page until you respect our new iPad2, but ok, we can crank out a press release if it saves us from the vorpal blade."
DOWN CURTAIN // insert character development and pathos before first rehearsal
(It's also possible - even likely - that Apple already knew about the bug. Apple does quite a lot of QA and every product ever shipped has had known bugs that weren't deemed so important as to postpone release.)
Some are located more than 100 miles away because the database contains every location ever logged. Despite the fact that hotspots and cell towers over the horizon cannot play a role in accurately determining your location, Apple's response is intended to create the impression that they play such a role and thus justify permanent storage.
Furthermore, short of magic, there is no way to send a relevant subset of the crowd sourced data to an iPhone without first knowing both the location of the iPhone and its unique identity.
>"The entire crowd-sourced database is too big to store on an iPhone, so we download an appropriate subset (cache) onto each iPhone. This cache is protected but not encrypted, and is backed up in iTunes whenever you back up your iPhone. The backup is encrypted or not, depending on the user settings in iTunes. The location data that researchers are seeing on the iPhone is not the past or present location of the iPhone, but rather the locations of Wi-Fi hotspots and cell towers surrounding the iPhone’s location, which can be more than one hundred miles away from the iPhone."
Apple is trying to create the impression that storing the data from which location can be triangulated is somehow significantly different from storing the actual location and again creating misdirection with the reference to "more than one hundred miles away from the iPhone."
>"5. Can Apple locate me based on my geo-tagged Wi-Fi hotspot and cell tower data? No. This data is sent to Apple in an anonymous and encrypted form. Apple cannot identify the source of this data."
In an interesting shift of language, Apple's answer is technically about the person's location rather than the location of the iPhone and it could be argued that in this context "source" refers to the person using the iPhone rather than the identity of the iPhone. Given that "cannot" rather than "do not" is used, the limitation does not correlate with something in an algorithm since an algorithm can be changed to identify the specific iPhone.
If you consider recording the id of the cell tower the phone is connected to magic..
The most charitable case would be that Apple only tracks location based on the cell tower to which the iPhone is connected. The worst case is that it tracks location based on every hotspot and celltower the iPhone sees.
Since the most charitable case would produce the least predictive power when selecting a relevant subset of the crowd sourced data and the worst tracking case would produce the most predictive power - and given the level of detail reported to be stored on the iPhone is consistent with the worst case and less consistent with the most charitable case - the worst case scenario regarding tracking would appear to be more likely.
+(DBConsolidated) subsetForDevice:(NSArray *)visibleTowers:(NSArray *)visibleHotSpots
"Anonymous" is pretty slippery and there is no agreed upon technical definition (just ask EFF), but it literally means "not identified by name" so one could argue that sending tracking data based on your device serial number, IP address, phone number, location, contacts list, etc. is still in an "anonymous form." [somewhat similar to debates about the technical meaning of "open"].
I'll add that the information you are proposing to send to Apple is enough to clearly identify your location - or rather the location of the iPhone, and short of Apple using Tor or a similar approach to making the message's route through the network untraceable, the data in "anonymous form" can likely be disanonymized.
Alright...
>Apple is now collecting anonymous traffic data to build a crowd-sourced traffic database with the goal of providing iPhone users an improved traffic service in the next couple of years.
...... The collection of anonymous traffic data involves tracking your location to determine what road you're on and what speed you're going. They can't even get their story straight.
So, it's logging your rough location...
Wow!
Great, thanks a lot! Now I no longer have the option of viewing what law enforcement will be able to get anyways. Nor will I have access to what essentially was a pretty neat database to look through.
I guess what I really meant was I'm happier knowing what they're keeping. Future versions may add other features of interest & we'll never know.
I wonder what this story would have been like if this database was originally encrypted & then it was discovered. "Apple attempts to hide location tracker on your iPhone via encryption!"
the whole q&a is a simple game words to trick users. `apple is not tracking ur location` but `your approximate location is downloaded, timestamped and stored on ur mobile phone due to a bug`
this whole wording is can't even trick a child.
look at the result.
Apple is now collecting anonymous traffic data to build a crowd-sourced traffic database with the goal of providing iPhone users an improved traffic service in the next couple of years.
....
Let me guess...Where, all roads lead to Apple?
EDIT: I posted this fully knowing it will be unpopular. But reading the press release gave me the feeling that Apple was using a sleight of hands by turning the public's focus from the privacy and security issues this incident has amplified and brought to the public's view, and instead is saying "Gee..We were doing y'all a favor by building a better maps app, and now you come along and screwed that up". Security is not an afterthought people. Hasn't the Sony fiasco that is still unraveling taught us anything?