Using Spotlight from the OS X Commandline | Hacker News Reader

Using Spotlight from the OS X Commandline

0xfe.blogspot.com

113 pointseloisius31 comments

I am very fond of the Mac, but the developers of Spotlight and the recent Versions feature of OS X Lion have not given proper thought to users' security and privacy.

Spotlight peeks inside files to create an index, and Version-supported apps maintain a changelog of file diffs. These OS X features save data in hidden locations, separate from the file.

This means that:

(a) it is very difficult to delete a file and be certain it's really gone,

(b) you can't have a public area, a private area, and a highly confidential area on the same system; it's all commingled to the same level of privacy,

(d) pieces of an encrypted file (if you use TrueCrypt for example) might get saved unencrypted in Spotlight's and Version's hidden locations,

I could go on, but you get the idea. It breaks down the notion of a file having a distinct level of security or privacy, and doesn't replace it with a better notion.

We already had the following problems:

(1) deleting a file doesn't really remove it on any major OS, including OS X, unless you use a special shredding program, and these days even shredding may be ineffective on solid-state drives because of wear-leveling and caching,

(2) the swap space can contain pieces of a file,

(3) temporary files created by editors, word processors, and other apps might retain parts of file.

Now Spotlight and Versions make the problem even worse for Mac users who care strongly about privacy and security.

We're getting to the point--or probably well past the point--where you have to maintain at least 2 or 3 separate systems: one for public work, another for personal or private work, and a third for highly confidential work.

That seems like an excellent trade-off to me, a prime example of how Apple is great at picking the right trade-offs.

All what you said is completely irrelevant for the vast majority of people. So Apple doesn’t trade it off against the increased convenience both Spotlight and Versions give you.

irrelevant for the vast majority of people

In real life people do think of pieces of paper as having different levels of security or privacy. Some pieces of paper you can leave out on the coffee table, others get locked in a desk, and still others in a safety-deposit box.

Also, as separate point, your use of the word "trade-off" assumes that there is no alternative other than giving the user some complex privacy-aware UI or complicated procedures.

That comparison doesn’t work. At all.

Why not? It shows people are very aware of privacy and encryption in their day to day routine.

At least there's full disk encryption with FileVault, if you want encryption.

Yes, and under OS X Lion, FileVault2 is apparently a complete rewrite of the earlier FileVault, and it appears to be solid. TrueCrypt is also an excellent (and multi-platform) alternative.

But using whole-disk encryption doesn't invalidate my comments about Spotlight and Versions. When you've logged in (i.e., mounted the encrypted volume), Spotlight and Versions will still be storing pieces of yours files in hidden locations, you still can't truly delete a file without a lot of know-how, etc.

You can easily exclude folders or disks from spotlight indexing.

System Preferences -> Spotlight -> privacy tab

to overwrite then delete the spotlight indices, disable spotlight then overwrite and delete:

   $ sudo srm --simple -rf /.Spotlight-V100/

There's also whole disk encryption (unfortunately not particularly robust -- I don't understand the reason but something about how the login password is stored seems to make this somewhat weak to eg the government) built in, as well as encrypted disk images and truecrypt.

I do know how to turn off Spotlight, but thank you.

My broader point is that it is becoming impossible to truly delete a file or maintain separation between different parts of your data unless you are very technical and meticulous.

How many other features are there in OS X like Spotlight or Versions that we need to be aware of?

The OS should be designed so that we don't need to be constantly on guard for these privacy gotchas.

I would argue that disabling Spotlight indexing for a particular folder isn't that technical, the UI for it is very simple. Of course, realising that you should do this perhaps requires a more technical mind! Overall though the point remains (swap, SSDs, etc), and is quite scary.

I misunderstood. In that case, yes, I totally agree. I'd suggest whole disk encryption but users will hate it, since if they forget their password the data is really gone. For reals.

> There's also whole disk encryption (unfortunately not particularly robust -- I don't understand the reason but something about how the login password is stored seems to make this somewhat weak to eg the government)

As far as I understand it, with FileVault 2:

- encryption is full disk AES128 (with optional AES256) with XTS.

- the AES key is apparently [1] stored in the keychain on the recovery partiton, which is itself encrypted with 3DES, and unlocked with the login passwords. The login passwords are not stored anywhere, they are used as the encryption passphrase: decryption failure means that the provided password is wrong.

People have been able to retrieve the AES key with DMA attacks via special Firewire devices on a running system in less than a hour (I guess they could have used the PCIe slot, and maybe Thunderbolt) by dumping and scanning the memory, and this impacts just about any system out there, not just Macs and FileVault, but TrueCrypt also [0].

[1] http://support.apple.com/kb/HT5077

[0] http://www.lostpassword.com/hdd-decryption.htm

IIRC the Spotlight metadata is stored per-volume, so if you for example mount an encrypted DMG or external disk the metadata for files on that volume are stored on the same volume. This way you can have separate volumes for things with different classifications, and information doesn't leak from one to the other via Spotlight.

If you strongly need full privacy separation between contexts, set up a virtual machine for your confidential work, and encrypt that. Then you can monitor what crosses the security boundary yourself, and deleting the VM means the data and any possible fragments of that data generated by nosy operating system features are gone.

Side note: Check out Finder's advanced preferences for 'Empty Trash Securely'. Additionally, I believe the swap space is now encrypted by default on Lion (at least for laptops), even if you don't have full disk encryption enabled.

Don't worry. Apple will solve that problem by completely hiding the filesystem from you :D.

It's nice to have access to such UI components from the command line.

One of my favorite is pbpaste/pbcopy.

What's yours?

$ softwareupdate -i -a [install, all] as it avoids the (slow) UI and you can schedule it if you want to.

Not sure if it'll carry across to ML, what with updates going through the Mac App Store there.

One downside worth mentioning is that you can't read the full description of each available updates, like you can when you click on an item in the GUI version.

Indeed. Moreover, I only want to update my jdk when Im ready and have some free time to repair my jdk 1.5 symlinks once the update completes.

    softwareupdate -d -a

downloads all updates to /Library/Updates. For each update, you get a .pkg file to install and a metadata file that includes this description.

`security` lets you manipulate your Keychains.

`say` lets you do text-to-speech in command line.

say is pretty useful and fun. When I'm compiling or just telling the Terminal to do something tedious, I add a say command at the end and it will notify me when it's done.

For example, if I want to install mercurial and git with homebrew:

brew install git mercurial; say "The task is finished, Master."

As an aside, in Lion at least, you can download high-quality voices for dozens of languages in Speech Preferences (including the ones Apple uses for Siri in several countries).

I used to SSH into my wife's Mac and asked her to come over if we are in different rooms :)

A more silent option is to print the bell character to the terminal, ie.: sleep 5; printf '\a'

This will cause the terminal to bounce and add a little badge with the number of tasks that require your attention if it's out of focus :)

`open` is very convenient to… open things (see `man open`).

Tip: `open .` opens the current directory in the Finder.

Another tip:

    open vnc://[user@]host[:port]

connects as user to host on port with Screen Sharing, and, more generally, "open URL" works for any URL scheme registered with Launch Services, including, but not limited to, the obvious ones.

    osascript

of course: it lets you control the UI of Cocoa applications, and even more for scriptable ones:

    osascript -e 'tell application "Safari" to close every document ¬
                  whose url contains "news.ycombinator.com"'

(¬, option-(lowercase L), is the AppleScript line continuation character)

Seems like

alias locate='mdfind'

is in order (on my system).

mdfind is basically orthogonal to locate (doesn't match individual files inside packages; handles wildcards differently), but it does produce find(1)-compatible output, which is nice:

http://jasomill.at/mdfind-dired.el

On my mac command+space switches the input mapping. Just sayin'

I'm interested in accessing this programmatically, of course. I've been thinking a lot about "Memex" lately, and the Spotlight index could be a very good engine for that.

Reply on news.ycombinator.com