List of URLs checked by Twitter for its app targeting | Hacker News Reader

List of URLs checked by Twitter for its app targeting

gist.github.com

112 pointsmartinml57 comments

This makes me ask questions that Twitter shouldn't be making me ask.

I don't presume to stop them from doing whatever they're permitted to do, so instead I ask myself:

"Should I uninstall the apps mentioned as their presence leaks information about me, or should I uninstall Twitter for spying on my device?"

Initially I thought that I use Twitter, so that must be high value... I'll delete the other apps. Then, looking through the list it occurs to me that as this expands I'd need to uninstall everything else except Twitter to render their spying useless.

Now I feel that the best solution is a very simple one: Uninstall Twitter and use the web version instead.

I guess that's not the outcome they want to be steering people towards.

Edit: The web version feels like a very old iOS app. This isn't necessarily a bad thing, it's fast and snappy.

Other Twitter native clients are available - at the moment. I'd suggest some but because of the API key limit any one that gets too popular will eventually be banned for new users. This killed the excellent multi-column Metrotwit recently, for example.

I've found myself doing this for most social networking on my phone. I only wish to give most of these apps a subset of the permissions and data they seek, and I honestly don't want to be bothered rooting and installing one of those permission gatekeeper apps, so I just use the mobile site. My Nexus 5 is sufficiently fast, and I have unlimited unthrottled data anyway, so why not?

You can opt out from the Twitter App. Or enable "Limit Ad Tracking".

Yes. but what enrages me is that this can only be done from the app. after you've installed (/upgraded) it and they have scanned your device.

My current twitter install on android doesn't have the feature to disable it yet, which means that i have to follow an upgrade path that includes updating -> toggling flight mode -> opening the app (if that works) and then toggling the setting.

I cannot disable it from the web interface, or this is put under a very obtrusive description.

Limit Ad tracking is an iOS system setting. For Android you can set "Opt out of interest based-ads" (Accounts -> Google -> Ads). You can set both before upgrading the app and no scan will take place.

I'm pretty sure that Android setting opts you out of Google-interest based ads, ie the same setting you can change via https://www.google.com/settings/u/0/ads

I doubt that changes anything for Twitter. AFAIK it doesn't stop an application doing scanning like this.

It doesn't forcefully stop apps to do this but Twitter reads and obeys that setting.

You can always add custom CSS-styling to the mix, if you don’t want to use third-party clients.

Maybe you should just stop using the internet altogether if you are this worried about being "spied" on.

Third party Twitter clients.

Tweetbot: https://gist.github.com/genadyo/295a5e8f0d743f57137f#file-gi...

Twitterrific: https://gist.github.com/genadyo/295a5e8f0d743f57137f#file-gi...

Twitterfon: https://gist.github.com/genadyo/295a5e8f0d743f57137f#file-gi...

Echofon: https://gist.github.com/genadyo/295a5e8f0d743f57137f#file-gi...

Am sure there are others.

Thanks for sharing my list :)

Under Android, it's possible to get a list of all installed applications by querying the packageManager: https://developer.android.com/reference/android/content/pm/P..., int), rather than "brute-forcing" known URL schemes. This doesn't require any special permissions.

Doing it this way means it'll work on iOS and Android, and it'll continue to work if the installed applications API is ever removed or blocked by a permission. Using the URL scheme essentially can't be blocked because it's necessary for inter-app comms.

Using the URL scheme essentially can't be blocked because it's necessary for inter-app comms.

It could be blocked via a permission as well (and just look like the target app is not installed.)

If you read the details closely of what Twitter is doing here, it's even worse than I had imagined:

From their help center [https://support.twitter.com/articles/20172069]

How will I know this feature is turned on for my account? We will notify you about this feature being turned on for your account by showing a prompt letting you know that to help tailor your experience, Twitter uses the apps on your device. Until you see this prompt, this setting is turned off and we are not collecting a list of your apps.

So, they collect the data first, and then they prompt the user telling them what they have done. This is the opposite of privacy friendly.

How do I turn this feature off and remove my data from Twitter?

Note carefully the overloaded meaning of the word Twitter here. Do they mean the Twitter app, or the Twitter service, or Twitter as a company? Grammatically and meaning-wise, the first one is the only one that makes sense. Which is very alarming...

Because it means, after they "remove" your data from the app, they still have your data. Or does it? It's not completely clear, which is part of the problem. The help text reads one way (no worries, you can delete your data) on a quick reading, but a completely different way on a careful reading.

You can easily adjust the setting that allows Twitter to collect a list of apps on your mobile device. Once you turn off the setting, we will remove your app graph data from Twitter and stop future collection.

Again, one has to wonder what they mean by "remove your app graph data from Twitter." Call me paranoid but to me this reads like weasel words and they still keep a copy of your data, just not on Twitter, whatever they mean by that.

So to recap, the really bad known thing here is they collect the data first, and ask permission later. The possibly really bad unknown thing is maybe they keep the data even after you think you are asking them to get rid of it, while trying to make it appear that they don't.

The problems are:

1. There is no API abuse or sneakiness happening here. They are just using a known, unrestricted API: https://developer.apple.com/library/ios/Documentation/UIKit/...:

2. Perhaps the API should be restricted?

I don't know how I feel about it. I don't know if I care if Twitter knows what other apps I have installed. This API is what allows Tweetbot to open links in Chrome, and I'd hate for that to disappear.

Maybe Apple can update the API to prompt the user and store that permission for each app?

I also feel that I don't care much if Twitter has my app list (partial app list, that is, because not all apps have URLs).

But how you and I feel isn't the point. Each user will have their own feelings about their privacy, and they should have the ability to control their own information in the way they prefer, with prior consent and opt-in, not opt-out once the data is already taken.

Yes, I'm sure it's very important that they know whether I have the Dunkin Donuts app installed (dunkindonuts://)

What am I looking at?

Twitter abuses an ios api to check if they can open a url with a specific app scheme. If the api returns true it means you have the app installed. You are looking at the list of apps that twitter checks.

This may be useful:

https://news.ycombinator.com/item?id=8665581

http://www.wired.co.uk/news/archive/2014-11/27/twitter-ads

https://support.twitter.com/articles/20172069

I'm a bit out of the loop on corporate data gathering. Why is Twitter collecting this data?

This is a sincere question, from their point of view what exactly are they doing and why?

It's for better ad targeting http://blogs.wsj.com/cmo/2014/11/26/twitter-is-tracking-user...

My guess would be that ads for app installs are hot these days and this is probably the best signal for it.

I am sorry for my ignorance, but can somebody explain what this list means? How are they checking for those URLS? Are they monitoring all URLs I visit in my device? Should I be concerned?

They check these urls to see if you have the associated app installed.

Legitimate uses would be things like checking if you have Google Chrome installed and giving you the option to open urls in Chrome instead of Safari.

Assuming this list is accurate, many people might wonder why they need to know if you have Angry Birds Star Wars II installed.

having worked for a company on the whitelist (and worked with Twitter directly for "whitelisting" scenarios), i would have to guess this is for proper deep linking integration.

that is, if Twitter links to "App X", Twitter needs to know if it can open App X directly or if it needs to direct the user to some website for App X instead.

i'd blame Apple for making this a notorious pain in the ass before i blamed Twitter for trying to fix it.

Could you expand on what aspect of deep linking requires Apple whitelisting?

I don't think so. I'm pretty sure from iOS you can use canOpenURL to determine whether an app exists. This is probably the method they're using to gather data.

Now I'm done with Twitter. Its a low value noise machine anyway.

I recently deleted my twitter account and I don't miss it a bit. Slowly getting there with Facebook but it's harder because I use Messenger a lot.

Probably dumb question, but why is every app registering their own URL scheme? Is that an iOS requirement?

It is mainly for deep-linking from urls in other apps/sites. Primarily so apps can reenter previous states rather than opening on the app, in some cases to push data to the app.

Example: Game app has buttons on a website to play specific levels with some special powerup, so you can link to it just like a website but in the app play the level and bypass the main screen etc.

It is a bit like DNS hosts within the devices but the only problem is it is not standard and there is no listing so it is more like old school file type or port claiming, there may be clashes/name collisions.

It's one of the few ways apps can interact with each other under iOS— by telling the system that they can handle particular URL schemes, and having other applications ask the system to open URLs with that scheme. It's kind of ridiculous, but it's the de-facto-standard convention for inter-application communication on iOS.

One common reason is that it is a requirement to do the fast app switching Facebook auth, which is why you see many URL schemes in that list in the form of fb[facebook app id][optional suffix]://

This is so that after auth the Facebook app has a URL to fast app switch back to and handle the authentication result.

It's not a requirement. Apps can use these to send someone to another app, along with optional information. For example, it's how you used to login via Facebook (before it was built into iOS).

One wonders if I have a large quantity of ad bucks to spend, and I'm willing to sign a NDA, if I could "target users who have installed X, Y, and Z competitors' apps?" (ditto for FB)

Will Apple phase out URL schemes now that we have extensions ?

But the thing is that URL schemes are very convenient in some cases.

One possible solution could be that you have to include all the URLs your app intends to open in its plist file. So if you're going to list hundreds then they can go ahead and reject those apps. But this wouldn't provide perfect privacy.

So my guess is that URL schemes will be yanked soon and all developers will be forced to use extensions for inter-app communication.

Not so sure URL schemes are that high on the chopping block. Using custom URL schemes is a suggested practice for Today extensions to launch their containing app with contextual information[1]

[1] - https://developer.apple.com/library/ios/documentation/Founda...:

I commented in one of the other threads, and said,

I'm really curious what the Apple AppStore review team has to say about this.

Well how should they know? They have a sort of binary parser that searches for unauthorized use of some (non-public) APIs, but I don't see how this use of an authorized API (-canOpenURL) would be problematic.

They won't be able to see what is going on unless the start the app under Instruments and document how many calls to this function are made.

Well, first of all, they probably know by now :) So I'm wondering if that will change anything.

I would also think they are doing run time profiling, if only to catch private API usage via NSSelectorFromString. Maybe they can add a test for excessive canOpenURL calls now.

Or maybe they don't care about this and more apps will do this kind of snooping going forward.

Deleted comment

Ad Targeting usually implies some kind if data "leaking" to get that targeted information. They are building that profile somehow and I think that's the part people mostly don't like.

Twitter seems to honor "Limit Ad Tracking". At least I don't see the option and I have limit ad tracking turned on. If that is the case, then I don't think I'm very concerned.

Should I be flattered that two of my apps are on there?

Totally! Why wouldn't you be?

Next thing you know, Apple limits the number of URLScheme checks any given app is able to do within a single session... should be interesting!

I read "petrescuesaga" as "Petres Cuesaga", the possible name of a developer. A Google search proved my brain wrong :)

What are all those fb[0-9]+://-schemes?

This is URL scheme an app needs to add to their info.plist in order to integrate the Facebook SDK. The number after fb is the ID of their page/app on Facebook.

In particular, I think it's used for switching apps - your app switches over to the facebook app (if installed) using fb:// perhaps, and if the user logs in and approves the fb-app, the facebook app will ask iOS to open "fbYOURNUMBER://", bouncing the user back to the app.

Pretty interesting that it includes our Thermo app as well.

My Lenovo phone comes with integrated firewall out of factory... Just saying.

Has nothing to do with this issue. Custom url handlers respond inside the device.

Has it been confirmed that these are being used for ad targeting? Or is this just a frenzy of indignation based on speculation?

[Update: OK, wow, yes I see, Twitter themselves announced the tracking: https://support.twitter.com/articles/20172069]

There are legitimate uses for this information in an app, such as for when a user is given the chance to tweet about a high score from a game, for example, and the app URL for the game could be used to get them back into the game app after they finish in the Twitter app.

I'm not saying this is the case (I don't know). It would be interesting to see whether all the apps in the list have some way that they interact with the Twitter app.

Reply on news.ycombinator.com