.Trashes, .fseventsd, and .Spotlight-V100 | Hacker News Reader

.Trashes, .fseventsd, and .Spotlight-V100

blog.hostilefork.com

89 pointspmoriarty66 comments

A few little arguments in favor of this practice:

1. You plug a USB device into one Mac. It asks you if you want to use it for Time Machine. You say no. Now, no other Mac will ask you this question again—because the preference has been stored on the drive, instead of on the computer.

2. You create a folder with an app you've written in it, and customize it with a cute "drag [this] onto [shortcut to Applications folder]" background, centering the icons on the background positions and sizing the window to perfectly fit the representation. You then copy this folder to another Mac (or any equivalent process, such as converting it into a DMG disk image and having someone download it.) They see exactly what you saw.

3. You keep an external hard disk that you frequently move back-and-forth between two Macs. The first time you plugged it into one of the Macs, it was indexed. Each time you modify it on either Mac, that index gets updated. The most recently created files are the most likely to be the ones you'll want to find through Spotlight when you plug the disk into the other Mac—but if the index was machine-local rather than disk-local, the new Mac wouldn't know about the new files yet, and would have to finish re-indexing to find them. (This is, of course, what happens when you modify the disk on a Windows computer, and it sucks. We really need a "metadata index API" to start being an expected offering of a filesystem, such that all OSes interacting with the filesystem can read and update it!)

On the other hand, Windows' Thumbs.db is exactly the sort of thing you don't want cached on the media. It'd be a great innovation if, say, cameras could pre-generate Thumbs.db files so Windows could immediately show the user what was on an SD card the first time it got plugged in... but cameras don't, because Thumbs.db is a random proprietary format. So, instead, it's only useful in the same situation that the Spotlight index is: moving a drive full of photos back and forth between two computers. Which might have separate screen DPI, and thus be set to show different thumbnails, meaning new thumbnails will get generated anyway.

I've always wonders why apple doesn't have some sort of .app.gz "special". Why does DMG have any reason to exist?

Even if you really really think you need to ship a readme (Hint: No one reads them, and you don't) a zip is simpler and opens quicker.

> Why does DMG have any reason to exist?

History and backwards compatibility.

The format itself also does a bit more than just a file archive, as it's a disk image format. You can do things like style the background and so forth.

1. Because a DMG can hold an HFS+ filesystem, and an HFS+ filesystem can represent files with resource forks. So, backward compatibility.

2. Because a DMG actually is a disk image, not a partition image—and therefore can represent a collection of volumes. If I recall, OSX's current "clean" install process effectively just takes a DMG containing {EFI, OS Base, Full Restore Image} volumes and uses the functionality of the "Erase and Overwrite with a DMG" option in Disk Utility to replace your disk's contents with it.

A gzip could represent HFS+ forks and attributes the same way the article is discussing - AppleDouble files and .DS_Store files. In fact, this is exactly what is done with zip files you create in the Finder.

Anyway, there's no way Apple is going to make improvements to the manual app install process anymore now that they have the App Store.

tar (which is what you mean) could do this, but until OSX Macs didn't ship with tar. DMG (technically the UDIF file format) came about as a continuation of .img files (the NDIF file format); there had already been built up some ecosystem around .img and .smi files and Apple wasn't going to suddenly switch to thinking in terms of tar files when they didn't have to.

Plus, a UDIF image can be simply passed to the kernel as a loopback image for a block device, and the files on it will then be discovered as plain files. For the same to work with tar or zip files, Apple would have had to introduce either something like the concept of Plan9/FUSE-like server processes that translate arbitrary backing stores into filesystem mounts; or break the universal abstraction of a filesystem by creating a higher-level shell namespace that could "see into" archives (like Windows and GNOME do) making OSX's Unix integration much less useful.

Mainly, though, it's that zip and tar are archival file formats. They're suboptimal for doing random-access updates on their contents, which is an important use-case of disk image formats. DMGs exist on OSX for the same reason that VHD(X)s exist on Windows: it turns out they're just a pretty great and flexible format for doing a lot of things OSes want to do.

>why apple doesn't have some sort of .app.gz "special"

In addition to other fine explanations provided, it did. The defacto standard was ".sit.hqx" -- BinHexed Stuffit archive, both third-party tools that survived into the OS X era. Disk Images came about when Apple finally realized the pain of delivering Mac apps over the internet.

Also, since I'm posting, OS X should really set the H attribute on these dot files when writing to a FAT drive, rather than relying on Unix shell convention.

Ah yes, good old .sit.hqx—compress something and then bloat it back up by converting it to ASCII. Luckily macbinary (.bin) took over at some point so that resource forks could be encoded without resorting to ASCII.

And you're right. Apple should definitely set the H bit on FAT volumes. Then no one would even notice.

Or, OS X could wait until the appropriate time to offer you the option of messing with your disk.

1. Don't ask about using the disk for Time Machine. When the user goes to Time Machine because they want to use Time Machine, then have a button in the sidebar "Use drive blah for Time Machine" for any non-time-machine external drives present.

2. Have a button at the top of a finder window in which icons have been re-arranged saying "preserve layout in this folder" (which you only have to click the first time, of course it remembers along with the layout). (Also, make it more convenient to default all folders/finders to list-with-details view, for people who have any idea what they're doing.)

3. When the user opens Spotlight, have a button in the sidebar "index this drive for Spotlight" for any un-indexed drives present.

In all these cases, it's still super duper easy to use all these things, without always bothering people who don't want these things.

> then have a button

> Have a button

> have a button

That's your problem right there.

Ugh, OS X pseudo simplicity is the worst. Want to align the dock in a corner? Want to tab through dialog options? There's an arcane terminal command for that. The list goes on, I have these to on top of my head.

At least the first two problems are the exception not the rule.

I want most of my USB-Drives not to be used as a Time Machine, therefore it should be the rule: Drives are not used for Time Machine unless noted otherwise. Since there's a pref pane for it, it wouldn't even need an extra button.

Most user's will not prepare the layout of the files and folders in some fancy way, so just show them arbitrarily, unless otherwise noted. Just hide the option in the "Get Info" dialog.

Drive Indexing options could be hidden in the "Get Info" dialog, but I'd be fine with keeping it enabled by default.

As a Kubuntu user and dedicated nerd, I fully agree. But as somebody who has spent some time trying to make user interfaces understandable to people, I'm a little more pragmatic and can see that pseudo simplicity still yields good results.

The Time Machine thing is pure marketing, of course. It drives conversions. That's its own brand of terrible, but even THEN you could argue that pushing people to do backups in whatever way isn't necessarily pure evil.

I'm not sure how Time Machine is pure marketing? It seems kind of useful.

It is useful but it's proprietary and could possibly leverage their own backup drives. Personally, I prefer the explanation that it's there to drive users to create backups. Maybe because Apple knows how reliable HFS+ is ;)

I think skore's point is right though: reminding users to make backups is the lesser evil.

I actually meant both - You CAN consider it the lesser evil while it also has the nice side effect of clearing the market (for simple backup in OSX) from competition as early as possible.

I guess tim333 means "asking if a drive should be used for Time Machine when it's first plugged" is pure marketing, which seems fair to me.

That’s non-sensical.

The central problem with backups is that no one makes them but everyone should, without a question. That’s an attempt to solve that problem, even going so far as to purposefully annoy people! (Remember, everything is a trade-off and usually you will never be able to make everyone always happy.) This dialog box is the furthest thing from a marketing gimmick I can imagine. It’s comically far removed from that. (And, I think, exactly the right move. Yeah, people who know about backups already probably will not use Time Machine – but for everyone else that is pretty great!)

Would you dare an (unverifiable) guess at the percentage of people who plugged in a USB drive to copy some files and were enlightened to use it as a backup unit by the Time Machine dialog?

If I had to, gun to the head, I'd say that no one ever was.

I seem to remember that Windows pops up a baloon in the system tray every once in a while nagging you to configure the system for regular backups. That seems much more sensible.

Sorry, should have clarified - Marketing in the sense of: converting people into using an apple solution early, so they won't use one from another vendor later on.

I have no numbers to back me, but my impression is the vast majority of OS X users can't be bothered with doing backups even with a built-in Backup solution, much less a 3rd party one.

Time Machine isn't going to stop someone who places value on doing proper backups from doing them, and if it makes it easy enough for my mom do back up her files, then that's a net win — not marketing.

And hey, it only took me reminding her 5 times before she got the external drive and set up Time Machine — no family tech support involved.

That is this kind of thinking of engineers vs. regular user.

You know why Windows users are so prone to viruses and such? Because they learned to click "OK". They learned it, because they always got the question/offer if they want to proceed or not. But a regular user don't understand those technical details and you know, they don't _want_ to understand those details. So we learned them to click "OK".

The best UX for the regular user is any UI where regular users have less choices of technical details.

I said that already the other day: as an engineer myself, I were chatting with another engineer about a product they have done (device integrated into a car for special purposes). I said to him, that something feels wrong with the interface. Yes, all the required information is present. Yes, probably I myself would have designed the UI the same way. But over the time I've got some sense in terms of UX, which a product makes it compelling or just doing its job.

The same you see with smart home. For how long there are HVAC controls out in the field by Honeywell and others? The better one are all programmable in all regards. But only Nest, which removed a lot from the interface made the big wave.

That's not a good argument against my proposal. How is

Want to use this drive for Time Machine? [OK]

better than going into the time machine app / prefpane / whatever, and seeing [use drive WD-BOOK for time machine] on the side, non-modal? so people who don't know what they're doing don't have to click or even see anything?

UI/UX designers do a lot of really stupid stuff in the name of making things "simple" and "easy". Engineers who think they're more of a people person and know some UI/UX probably do it even more. Give up the gimmicks, the ridiculous space between elements. The material design, the flat-aqua design, it's all crap.

I've seen "regular users" fail to use all these things, and need my help find a way through the increasingly hidden and opaque interfaces to what they're trying to do. Usually because things that are supposed to "just work" instead "just don't work" and there's no way to fix it. Except for me. So interfaces designed for me would probably not be a bad idea ;)

Most of the users I know wouldn't know where to go to set up a Time Machine drive. The fact that it prompts them flat out is how they even got it set up in the first place.

Well said. This is good usability.

No need to contrive lame excuses.

It's just bad software design, plain and simple.

I believe the way you describe is the better way for users, but the problem seems to be about the .files littering the whole filesystem. Why not have a top-level .hidden folder, where everything is put in, for any OS ? You'd have the OSX's .DS_Store, Windows' Thumbs.db, and whatever Linux-specific dot-file you can have.

I kind of see a parallel with applications willing to put their stuff in ~/.myappconfig instead of the central ~/.config/myappconfig, and it bothers me as well.

> Thumbs.db is a random proprietary format

I may be wrong here, but it doesn't look to me that .fseventsd, .Trashes and .spotlight are open, standardized multi-platform formats.

Unless random proprietary format means "My mac doesn't read it".

".Trashes" is just a directory. There's no "format" to speak of…

Your point stands for the other files, though.

Just a side note, Thumbs.db are no longer created in visited folders since Vista. Mac OS seems to be the last OS that randomly trashes removable mass storage.

First thumbs.db was a WinXP only thing. Windows Vista and higher store the thumbnails on C:\ in the users folder!

But actually the thumbs.db file was great! The thumbnails were local next to the files. With WinNT 6+ the thumbnail store grows several GBs on C: for each user. If one moves a directory it has to reindex it. Attaching a USB removeable drive increases your central thumbnail file on C: even further. There is a pruning feature so if you don't access the USB storage the data is overwritten. Then the Windows Shell has to index Terabytes of external data again. Win7 has a bug that every now and then updating the central thumbnail store (and using Windows Security Essential) consumes more memory (16+ GB) than is available - it was clearly never tested with 100.000s of media files.

It doesn't help that Windows Explorer starting with Vista lost several convenient features like remember the file list style of every folder or using a "folder.jpg" file as folder preview picture instead of showing 4 random pictures. Win8.1/10 at least brought back the "up" button. WinXP had also a real toolbar that could be changed, now you need a custom shell extension.

Am I the only one who doesn't like file preview icons in my file manager? I always turn it off on Windows and OS X.

I'd be more inclined to berate Apple for this were it not for the amount of crap that Unix-style tools dump in my home directory.

I count 50 .-prefixed files or directories at present, some of which is from Mac developers who should know better (hello, Adobe, Dropbox), some is cryptically-named gunk from other tools (.vegas, .jmf), some various command line histories (bash, fcsh, mysql, psql, pry, sqlite, irb), and so on.

OS X is nicer than traditional Unix behaviour here - it puts it all in a well-organised ~/Library directory.

On Linux things should go in xdg dirs ~/.config ~/.local and ~/.cache instead of ~ but a lot of old unix software and sadly a decent amount of new software(even guis) don't follow the standard.

Even 50 is nothing. On my Linux machine:

    $ ls -d  ~/.* | wc -l
    342

I got the impression that he's not upset about all the dotfiles, he's upset that they put them on external media. Most unix programs will only create those in the users home, which is usually not external.

This accomplished two things:

1. I now know a few things I've wondered for a long time.

2. I have just been reminded of how little I really know, even about devices I use for most of my waking hours.

for the love of christ, it's OS X, not OS/X

To distinguish it from its predecessor, Mac OS IX?

Trolling is a art.

I want to see someone come out with an OS/2 skin of sorts for OS X, just to make everyone in the world that much more confused.

And I hope you're pronouncing that as "Oh Ess Ten" and not "Oh Ess Ecks" ;-)

So that .Spotlight-V100 store some kind of index, probably binary. I presume that any Mac reads it on any arbitrary USB drive it's plugged in (or, at least, when a search is triggered). By creating a specially-crafted, corrupted index I see a channel for denial of service attacks.

You can probably still abuse NSDictionary, causing HashDoS (https://gist.github.com/dchest/4467707). I did it with QuickLook a few years ago; don't know if they replaced a hash function there.

Hopefully users will push back; demanding to be given more options, and more control. We've gotten to a point where the role technology plays in our lives is too integral to keep letting that control slip away, one "harmless" feature at a time.

Amen to that. Unfortunately it seems the trend is towards hiding the filesystem/concept of files completely from the user and replacing it with app-specific isolated storage in a proprietary format, which is even worse than unsolicited filesystem modifications. (On the other hand, it could be argued that these modifications would become even less noticeable in such a system... which is good or bad depending on who you ask. I'm firmly on the latter side.)

A physical write-protect is something all removable storage should have.

I do this if I forget to add .DS_Store to .gitignore when working with git repos: find . -name .*DS_Store -delete

Just be careful. A coworker of mine did this recently and forgot the -name part. Accidentally deleted everything, including the .git directory. He sure wished he had pushed that day.

*that hour.

Surely you don't cruise through to the end of a day without pushing at least once before lunch? :(

I commit often, rewrite mercilessly, and thus I push to the big server infrequently. I do push frequently between local machines and servers. Mercurial Evolve makes propagating these edited commits across my local servers quite nice.

Anyways, besides the plug for hg, my point was that I would expect pushing could be a once-a-day kind of thing, whereas commits should be every five minutes. :-)

I agree. Pushing too frequently means you can't rebase and reorganize your commits - unless you each use separate repos.

    git config --global core.excludesfile ~/.config/git/ignore

    cat >> ~/.config/git/ignore
    .DS_Store
    .AppleDouble
    .LSOverride
    .AppleDB
    .AppleDesktop
    .apdisk
    .fseventsd
    .Trashes
    .Spotlight-V100
    ._*
    Icon
    Network Trash Folder
    Temporary Items
    Icon

you have missed

mkdir -p ~/.config/git

Why not .* ?

I can't think of a good reason to commit hidden files.

put stuff like this in your global gitignore and you will bever have to worry about these files again. Stuff like this does not belong in a projects gitignore anyway because it's system specific.

It seems to me, that if I insert a removable media that is formatted with one of the FAT filesystem variants, that I probably don't want those hidden files & folders created. As it's a sign that the ultimate destination probably isn't another Mac.

It is annoying when copying a movie to a USB, and having to empty the entire trash bin to fit it on the USB stick. I guess it does enforce you to be a bit cleaner.

I don't really care about .DS_store etc as they're hidden, and thumbs.db is just as bad.

> I don't really care about .DS_store etc as they're hidden

They ain't if you are on Windows.

That's really just silliness on the part of Apple; if they're creating dotfiles on a FAT32/NTFS/ExtFS disk, they should be setting the Hidden attribute for those same files. This is what e.g. Samba does automatically for network-attached FAT/NTFS folders on OSX/Linux clients, if I recall.

There was a patch submitted to the Darwin project to do this:

Https://lists.apple.com/archives/Darwin-kernel/2004/Jul/msg00120.html

I don't think it ever got reviewed though.

Thumbs.db doesn't end up in every zip file ever.

Eject for Windows (http://www011.upp.so-net.ne.jp/decafish/EjectForWindows/Ejec...)

Before OSX 10.10.2, apparently there was a "heartbleed" style info leak in spotlight where it "may save unexpected information to an external hard drive", according to http://support.apple.com/en-us/HT204244 (CVE-2014-8832). Probably related to this report: https://www.f-secure.com/weblog/archives/00002752.html

What happens when the drive is shared among Macs? Why isn't this a vector for exploitation?

Well, if you find a buffer overflow that can lead to code execution in spotlight (mds) or fseventsd then it very well could be a vector. Assuming no bugs in those (hah), then I'm not sure what you could exploit…

Never understood why they do this. Surely they should keep junk off removable drives, as they are the most likely to be used in other systems including ones that don't hide data from the user.

It's just nonsense that this metadata would be dropped as files scattered across the filesystem, rather than simply held in a single location on the host machine. They're a fragment of the OS, not of each directory.

Which would mean that those meta data is only available on this host and not anywhere.

The problem is the short coming of those file systems not to allow to store arbitrary meta data to any file.

That's exactly what I want though. "Folder display preferences" are a thing that pertain to exactly one instance of one operating system. If I want to share "folder displays" to a different computer (which seems rather unnecessary), that seems like an explicit action I might take between the two computers, and all that metadata should be a single bundled file that is transferred between machines. Not per-directory metadata that is bundled into my filesystem everywhere.

Reply on news.ycombinator.com