Your Mac Keeps A Log Of Your Downloads | Hacker News Reader

Your Mac Keeps A Log Of Your Downloads

macgasm.net

243 pointsfcukdigg86 comments

This is used to show where a program came from the first time you run it. For example, if I download iTerm 2 and then run it, I get the following warning:

http://i.imgur.com/IbUWj.png

Kind of brilliant, actually. Not the fix or the tracking method...just that I knew subconsciously that if OSX could know to warn me that I'm trying to open up a file downloaded from the internet (and which webpage it came from), then it must have some way of tracking it that supersedes the standard browser wipe. But I just never thought to look around for it.

Windows will warn you about files downloaded from the internet, too. It uses "Alternate Data Streams" feature of NTFS to mark the file. I suppose once you delete the file, the alternate data stream is gone as well. From what I read, alternate data streams were made for compatibility with a Mac filesystem!

And OS X could do the same thing with the resource fork if they wanted to instead of never forgetting.

Seems to me this would be a better way to do it, you're reading the resource fork for the file anyway, why also make a sqlite query as well?

No, the resource fork is dead for good reasons. (1) The resource fork is fragile and easily corrupted, like the Windows registry in the bad old days. (2) Four-character namespaces suck. (3) 24 MB limits suck. (4) Extended attributes and bundles are better in every conceivable way.

The resource fork was designed for a different world. (A) In the old days, there was a blurry line between OS and app, so there was no reason to make the resource fork "opaque". Without opacity, there is no graceful way to use it in a preemptive multitasking environment. (B) In the old days, pointers were 24 bits. The high 8 bits didn't go anywhere, so people used these high bits to store extra data. Resource forks make use of 24-bit quantities for this reason — which went out of favor when Macs with the 68020 were introduced in 1987. (C) The real feature of resource forks was to reduce the disk space usage and load times when you have lots of small pieces of data, but — data is bigger and we have better archive formats.

So yeah, extended attributes are the way to go.

There is also a maximum of one officially supported fork per file (i.e. in a single format), and even a single stored byte will occupy a full block. On HFS+ attributes are written as individual records directly to a per-filesystem b-tree, so wastage is no worse than any other metadata. HFS+ also has named forks but they were never a supported feature AFAIK

Deleting the file also deletes the stream (in as much as deleting a file marks it as OK to overwite). Scott Hanselman blogged several ways to remove streams in 2007 (http://www.hanselman.com/blog/RemovingSecurityFromDownloaded...).

Actually I think that warning comes from data stored in the xattr com.apple.metadata:kMDItemWhereFroms on the file itself. So this sqlite database must be for something else.

Hmm, it seems that the URL is taken from the database. If I delete the relevant row, the URL is no longer shown:

http://i.imgur.com/Ks03r.png

I'm not a Mac user but wouldn't the test therefore be to clear the DB entries and re-open the file. No prompt implies the link to the sqlite db and not the file attr.

The dialog only shows up when that xattr is set, the sqlite db has nothing to do with this security prompt. In fact all the prompt does is unset the xattr if you deem it ok.

Correct. There's a quarantine attribute.

The design decision of storing the application a file came from in file metadata, but the URL it came from in an sqlite database ... makes my head hurt.

So, I assume you have considered all the aspects of the problem to come to that conclusion right?

Not merely pondered upon it for 2 minutes and decided that Apple programmers were idiots for doing so, right?

You, or whomever did the screencap, must have a retina macbook. That dialog in the screencap looks huge on my computer.

Since we're OT, "whom" only works in objective case.

Which is to say, where you'd say him or her, rather than he or she.

"He, or whoever took that screenshot..." "Her, or whomever that screenshot was sent to..."

Nah, the warning comes from the extended attribute on the application folder/zip. It has nothing to do with the log.

You can disable the warning by a command line:

xattr -d com.apple.quarantine <path_to_your_program>

The extended attribute records that the file is quarantined, but it doesn't contain the detailed information about where the file came from. It just has an identifier that's used to look up in the database. For example, here's the quarantine info for a file in my Downloads folder: 0002;50e77ce1;Safari.app;BF047C49-2539-4536-A584-E69FDE328A2C. I don't know what the first two fields are, but the third is the app that downloaded it, and the forth is a UDID, which is presumably the identifier for the quarantine event.

Furthermore, even once a file is out of quarantine, the Get Info dialog can still tell you where it was downloaded from, precisely because of this database.

UUID, not UDID. The latter is an unique identifier for iOS devices.

Oops, typo. Too bad I can't edit my post anymore.

There is a reason why the marking of the file and the database containing the download information are separate. It is for personal privacy! If you copy or move the file or someone else runs whatever it is, the "quarantineness" is still there, but the personal information about where and when you downloaded it will not be accessible. The "taint" associated with the downloaded object will remain with the file, but the personal and potentially private information does not.

NAH, the warning is merely initiated by the extended attributed. The information in it comes from elsewhere (presumably the database).

This is amazing, whoever feels comfortable about it should band together and see what files are in common, or domains. I want to delete this yet I don't. This contains all (most?) of the files I have ever downloaded, those I thought I lost when clearing browsing data from 2008.

My first few files

* Symantec_Antivirus_Mac.dmg

* http://msdn01.e-academy.com|http://download.e-academy.com/do... (MSDN Alliance: Free Microsoft Software for Students)

* http://download2.vmware.com/software/fusion/VMware-Fusion-1....

* http://download.skype.com/macosx/Skype_2.7.0.330.dmg

Scary yes (The torrent files) but so much history to look at.

    sqlite3 ~/Library/Preferences/com.apple.LaunchServices.QuarantineEvents 'SELECT datetime(LSQuarantineTimeStamp + 978307200, "unixepoch") as LSQuarantineTimeStamp, LSQuarantineAgentName, LSQuarantineOriginURLString, LSQuarantineDataURLString from LSQuarantineEvent' | sort

Ordered by date

That's actually how I felt about the iPhone coordinates database (also in SQLite I believe)...I saved a copy of it before upgrading to the iOS that didn't have that "feature"

I guess this should help disabling. But I can't confirm as I'm not near my Mac test this.

  sudo launchctl -w com.apple.LaunchServices.QuarantineEvents[.plist]

Nice article, but unless the database is "VACUUM"ed after the "DELETE" the rows are still readable in the database file. This kinda defeats the purpose of the whole article b/c the user didn't gain any more privacy than he had before the deletion :)

can you share the command to do this?

sqlite3 ~/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV* 'vacuum'

Tried this, but I still see all of the files in the list.

The idea is that first you run the DELETE statement from TFA, which marks the records as deleted and "hides" them but doesn't completely "forget" them, then you run the VACUUM one, which should basically erase any trace of those records ever existing in the sqlite db.

Hm, they could also have tagged the file as "came from the Internet" in the filesystem metadata associated with the file itself. That way, the "came from the Internet" tag is only around as long as it needs to: for the lifetime of the file. Avoiding the privacy problem, but also more sensible for another reason:

If you'd rename or copy the file, the "came form the Internet" tag will remain or be copied with the file. With the sqlite database approach, either this association breaks, or you need to check the database every time a file is copied, moved or renamed and if it's in there, update the database, if you want to be able to track a file when it's copied or renamed.

I don't know if OSX has extra logic for this, or if they just allow the association to break. But with the metadata tagging approach, you only have to run the tagging logic when the file is downloaded (to set the tag) and when it's about to be executed (to check the tag), not with every other file-operation.

A strange choice, IMO: the sqlite approach makes it harder to achieve the intended goal because you need extra effort/logic required to track a file as it's copied, renamed or moved, while at the same time it makes it easier for an unintended goal: tracking users by keeping the information about the file around even when it's deleted and the "came from the Internet" warning is no longer useful.

As I mentioned in another comment, apple do tag "com.apple.quarantine" as an extended attribute on the program folder/zip itself. The download log has nothing to do with the warning dialog.

Am I the only one that disables this behavior by default?

defaults write com.apple.LaunchServices LSQuarantine -bool NO

Works like a charm...

It's also included in this sane defaults dotfile

https://github.com/mathiasbynens/dotfiles/blob/master/.osx#L...

That's not so much a "sane defaults dotfile" as a highly-opinionated personal defaults dotfile. I don't consider that a bad thing, but do at least give it the deserved appellation.

Words can have different meanings in a jargon than they do in the dictionary, but that does not make those meanings invalid. For instance, in computing "sane" means "I'm an overly opinionated git and everyone who disagrees with me is wrong."

That's brilliant -- I always wanted to remove this behavior, but didn't know where to start. Thank you.

Am I the only one who had nothing but updates for Adium Transmission and the occasional hot news rss from Apple in that file?

Not exactly shocking material. But thanks to OP for the heads up.

Files get a "quarantine flag" set on them as metadata when downloaded on OS X. Gatekeeper uses this (along with the developer signature) to check if an app is "safe" to open or not. Not sure where this sqlite database fits in, but it's very likely related to that.

The why is the Mac App Store exempt?

Trusted/controlled source.

Not just the Mac App Store, by the way. CDs or any other way of getting the file on the Mac except downloading are, too.

That all makes perfect sense. It doesn’t provide perfect protection, but it does provide sensible protection.

Strange, on my machine this file only contains the URL from when I installed Firefox and a bunch of URLs for Adium updates. Not any of the many other things I've downloaded...

Is it only listing things downloaded through Safari?

Same here. The only thing it shows I downloaded is Firefox.

It show's all mine, I use Chrome for everything.

Maybe Firefox downloads aren't included? Maybe it's watching the ~/Downloads folder? Can you confirm either?

I'm experiencing pretty much the exact same thing, with a few minor additions to your items: I also have iTerm2 in my list, some ADC files, and strangely some RSS xml files I have downloaded ages ago.

I can't see any of my downloads, the database exists but is empty on my system (10.6.8). (and I've sure downloaded a lot of files...)

You could try something like:

    $ sqlite3 -column ~/Library/Preferences/com.apple.LaunchServices.QuarantineEvents <<END|pbcopy
    > .mode tabs
    > .headers on
    > SELECT strftime('%Y-%m-%d %H:%M:%S', LSQuarantineTimestamp+ 978307200, 'unixepoch', 'localtime')  as date,
    > LSQuarantineAgentName as App,
    > LSQuarantineDataURLString as URL
    > FROM LSQuarantineEvent
    > --where LSQuarantineDataURLString like '%bankofamerica%'
    > order by LSQuarantineTimestamp;
    > END
    $

Then open your favorite spreadsheet program and paste in the results.

The 978307200 number corrects for the fact that Apple is using 2001-01-01, the year OS 10.0 was released, as its epoch.

I got way fewer rows than expected. Apparently, downloads by Firefox are not logged. When I screened for B of A downloads, I was puzzled to see only downloads in the past year or so. That's because I only recently started using Chrome to access that site.

And I suspect that if you had a virus scanner installed on windows every file it scanned would be logged somewhere depending on settings.

This post also fails to mention that this is specific to Safari at last check. I don't think Google Chrome integrates with this functionality.

I don't use Safari and it has all my downloads from Google Chrome.

I use a safari/firefox mix, all that's in that db for me is a short list of things I don't care about...

It does now:

    $ sqlite3 ~/Library/Preferences/com.apple.LaunchServices.QuarantineEventsV* 'select LSQuarantineAgentName from LSQuarantineEvent'|grep 'Google Chrome'|wc -l
          35

But firefox doesn't seem to be in the list, though I am using the Firefox Nightly so that has something to do with it. I have Adium (lot of sparkle update checks), Chrome (just used for testing), iChat and a couple other random one-offs.

Firefox seems to be "blacklisted" by Apple, see /System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Exceptions.plist :

                <key>org.mozilla.firefox</key>
                <dict>
                        <key>LSFileQuarantineEnabled</key>
                        <true/>

Nightly is org.mozilla.nightly, so that explains the grandparent's behaviour differing, since the Apple blacklist does not include it.

Chrome does integrate with this functionality.

Also, some apps doesn't even need to integrate, as the OS will override and auto-apply files for some blacklisted apps, see /System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Exceptions.plist for LSFileQuarantineEnabled

Files that aren't quarantines don't seem to end up in that log, unsurprisingly. No images, audio or video files, but applications and archives.

At the end of the post, the author mentions using Automator to run a script to delete this frequently. Alternatively, you can use launchd to run scripts on a Mac. http://nb.nathanamy.org/2012/07/schedule-jobs-using-launchd/

Actually, there are 2 versions of this file.

Depending on how long you have had your Mac, you will have both com.apple.LaunchServices.QuarantineEvents and com.apple.LaunchServices.QuarantineEventsV2

Naturally, both files will have to be cleansed.

I get 'Error: no such table: LSQuarantineEvent'

Same here. OSX 10.6.8

Ditto.

Install GNU+Linux, problem solved.

Only if you are paranoid enough to think that this is a problem. Of course, there are other problems with installing Linux that are ignored, but that's a topic for another discussion.

Guys, guys, watch out, I just noticed that my computer has a copy of all my files on it. Who gave Apple permission to do that? Privacy is dead!

Facts about files on disk that do not apply to the database entries described in the article:

1. A file on disk is the intended result of a download

2. A file is readily visible/alterable in a user interface provided for consumption of the same by the operating system

3. Removing a file means that, without serious effort, the information contained therein is gone

Funny, but sometimes you do want to clean up your laptop - for example, before crossing the US border.

I would be irked if I had taken steps to remove a downloaded file from my laptop, including secure deletion, only to leave evidence that I'd downloaded it in a sqlite database.

If you are worried about that kind of thing, I'd recommend doing that stuff in a VM with an encrypted file system. Then, when you delete it, you know everything is gone.

The article isn't even accusing Apple. Why do you feel the need to defend them?

I think this really can be a potential privacy issue. "Your computer having your files on it" is quite different:

a) You know that your files are on your computer

b) You know how to delete one of them in case you need to hide them from someone

What's perhaps more concerning is that Safari enters URLs in this database, even when Private Browsing is enabled. When you enable private browsing, Safari states:

"Safari can keep your browsing history private. When you turn on private browsing, Safari doesn’t remember the pages you visit, your search history, or your AutoFill information."

Well it does remember files you downloaded, even if you subsequently delete them.

It's really hard to tell what back doors proprietary operating system have created and you lose your rights to privacy as soon as you start using proprietary blob.

"Your Mac Keeps A Log Of Your Downloads"...

In your user account.

There's quite a difference right there. I thought it was some kind of "hidden" file (not unlike the GPS location which was saved by default on any iPhone with a GPS -- up to the latest iPhone!?) which had now been discovered.

It's just in your user account so it's no big deal.

Why is it different just because it's in your user account?

Everyone with access to your Mac can access it just as well, i assume. I guess it can also be read by applications, uploaded to the internet? I don't have a Mac so i don't know the restrictions but i would consider it privacy information that i would like to know if and where it's stored. There is a reason you can delete your browser history. Or your recently accessed files list.

> Everyone with access to your Mac can access it just as well, i assume

If they're logged on to your account, but not if they are using a different account.

>I guess it can also be read by applications, uploaded to the internet?

Not Mac App Store apps - they're quarantined.

> If they're logged on to your account, but not if they are using a different account.

So, they are encrypted or how is that guaranteed?

File permissions.

It's more than 'no big deal' because it does it irrespective of the Safari 'Private Browsing' feature.

"You should not be downloading anything that you don't want anybody else to know" Eric Schmidt would say

hahaha, good ole' Schmidt. What a gem.

He's created many a Schmidtstorm with his off-the-cuff remarks.

No, he wouldn't. "If you don't want Google to tell the government you've been searching for illegal things, you shouldn't tell Google you've been searching for illegal things" is what he'd try to say. But he'd mince the words and make himself sound evil, somehow. So maybe you're right.

Leave it to (taligent|barista) to turn an informational post about Apple into some off topic bitch about Google.

I hope he made this point clear to the North Korean people when he visited them.

What? I doubt it. I was pissed when I heard the full context of the quote you're referring to and saw what he actually meant. Here's what everyone likes to quote:

> "If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place"

Here's the quote in full context:

"I think judgment matters. If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place, but if you really need that kind of privacy, the reality is that search engines including Google do retain this information for some time, and it’s important, for example, that we are all subject in the United States to the Patriot Act. It is possible that, that information could be made available to the authorities."

Reply on news.ycombinator.com