Explainer: .DS_Store Files | Hacker News Reader

Explainer: .DS_Store Files

eclecticlight.co

136 pointsserhack_90 comments

These are arguably one of the smaller annoyances about apple when not using one myself (so, always) and they have become less relevant over the last years as sharing things via USB has become less common and I don't use dropbox or such with shared folders anymore. People who use git don't tend to check these in, when working with me at least, I make sure of that.

But it's a pretty darn good symbol for how Apple doesn't give a flying fuck about anyone's time/nerves, just dumping these on everyone without asking and letting us deal with their laziness.

Plus, Apple users then typically just rolled their eyes when I point out to them that I've now got all these files on my USB drive, thanks for nothing Apple. "Why do you even care?" Yeah, why do I? There's just no reason I should have to sort through these to keep them out of my systems, yet I don't want them there so no I have to, thanks apple... (this was before I got good on the terminal so it was actually quite annoying)

Bottom line: this still gets my blood boiling. Fuck apple, they're the new Microsoft in every way.

Theres also all `._.name` files MacOS spews all over network shares >:(

https://apple.stackexchange.com/questions/14980/why-are-dot-...

  defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true

Just tried this but it doesn't work. You might have to drag and drop some video (.mp4) or audio (.mp3) files to see that it still creates the ._ files

Also, FAT32/exFAT drives.

It’s just the same as desktop.ini files, it’s where they record the position of icons in storage locations that don’t allow the filesystem to store that information invisibly.

It’s not a big deal.

I’m still not sure why you care. You explained that you don’t want them on your ‘systems’ but you haven’t explained how they get there in the first place - you sound like you’re not an Apple user but these files are making it onto your USB drives and this is bad for you for some reason.

Wow, you sound exactly like the apple users I was referring to, except I can't tell whether you actually are one. :)

The point is that yes, "my usb drives" get polluted the second I give them to an apple user even if they just read files off them.

Same happens if an apple user accesses a writable network share on my NAS or my PC.

I care, because I constantly have to be on the look-out then to not accidentally add them to tar files, backups etc.

When I navigate my own file collections I just don't want these things lying around my folders everywhere, distracting me for zero benefit.

That's all work and mind capacity that apple happily, without asking, offloads onto me, because they won't bother _their_ users with it and because wtf am I supposed to do, refuse to interact with all apple systems?

If you do not open directory with Finder they do not appear. So why to use Finder anyway?

".DS_Store", or, "how to tell if a ZIP file was created by a macOS user".

"Desktop.ini" and "thumbs.db", or, "how to tell if a ZIP file was created by a Windows user".

Yes, but these files only get made, when:

- There is anything to create a thumbnail cache for

- User has applied custom settings like a folder icon or designation to the folder

".tar.gz" or how to tell the file was archived by a Linux user.

.tar.Z or how to tell the file was archived by a Real Man[tm].

I received some ZIP files from macOS users which contain a directory called __MACOSX .

I seem to recall that the __MACOSX folder is there to support resource forks? Something along those lines at any rate.

Yep. And since many files don’t have resource forks it’s kinda rare. But any directory you look at basically gets a .DS_STORE.

I remember hiding those from windows clients with samba, and hiding thumbs.db from mac clients over netatalk (and samba)

It looks like they store the extended attributes in the __MACOSX tree too. And everything you download from the web has extended attributes describing the original URL and page that linked to it. (With chrome/safari, I haven't checked firefox.)

It looks like the finder "compress" functionality will include __MACOSX, and the command line zip doesn't.

If you run `xattr -l` on something in your Downloads, you should see the kMDItemWhereFroms metadata. mdls shows it too, but also includes other data that is extracted from the file itself.

You can also search on that metadata:

    mdfind kMDItemWhereFroms:citeseerx

Another issue with zip files on OSX:

> Zip files can encode their file names in two ways: CP437, or unicode.

> Each operating system does it wrong, but in a different way. For instance, Mac OS encodes its zip files as unicode, but doesn't set bit 11 correctly, so Python (correctly) reads them as CP437, and garbles the non-ASCII characters in file names.

> I wrote a quick and dirty workaround for Mac OS archives: if the file doesn't exist, encode the name as CP437 and check again. I'll think of something more clever if I ever switch to another OS.

https://nicolasbouliane.com/projects/timeline

Woah, that explains a lot of weird bugs, actually! This is very useful information, thanks!

Yep, I write scripts to clean up zip files because the default behavior is so bad for cross-platform.

Could you please share it?

This is my script to “clean up” a typical macOS zip file. It assumes you’ve started by asking the Finder to “Compress” (which creates a zip file but one riddled with macOS-isms).

    #!/bin/bash
    
    zipfile=$1
    if [ "x$zipfile" = "x" ] ; then
      echo "$0: .zip file expected" >&2
      exit 1
    fi
    
    zip -d "${zipfile}" "__MACOSX*"
    zip -d "${zipfile}" ".DS_Store"
    zip -d "${zipfile}" "*/.DS_Store"
    unzip -l "${zipfile}" | sort -k 5

You probably want to test:

    if [ ! -f "$zipfile" ]

Instead of

    if [ "x$zipfile" = "x" ]

Thank you! I took your script, suggestions in this thread and added some of my sauce:

    #!/usr/bin/env bash

    set -exuo pipefail
        
    VERBOSE=false
    while getopts "v" arg; do
    case $arg in
        v) VERBOSE=true;;
    esac
    done
    shift $((OPTIND-1))


    zipfile=$1
    if [ ! -f "$zipfile" ] || [ ! "${zipfile##*.}" = "zip" ] ; then
        echo "$0: .zip file expected" >&2
        exit 1
    fi

    zip -d "${zipfile}" "__MACOSX*" ".DS_Store" "*/.DS_Store" "Thumbs.db" "*/Thumbs.db"

    if [ $VERBOSE = true ]; then
        unzip -l "${zipfile}" | sort -k 5
    fi

Processing the zip file three times over seems a bit excessive?

  zip -d "${zipfile}" "__MACOSX*" ".DS_Store" "*/.DS_Store"

True but performance wasn’t the goal (I tend to run it once on smallish files).

This is easier to extend if I see something new to exclude (or comment-out some rule).

BetterZip has an option for “clean” zips you can enable as the default for create.

Why does Windows not deal with hidden files correctly?

It's 2021. Hidden dot-files are not a new thing.

Hidden dot files were introduced as a bug and left to linger because they were kind of useful. It all started when someone tried to hide . and .. from the output of ls and messed up the if statement to only check if the first letter in the filename was a period instead of testing for the intended use cases. People then copied that behaviour around because it was a cool new trick, not because it was set up as a standard.

The Windows method, leveraging file attributes, is actually much cleaner in my opinion. You can set the hidden attribute in ZIP files and most tools do for OS specific files and folders, but I don't want my ZIP tool to put files on my file system that I don't get to see first so I always turn them on.

Windows does the same thing with desktop.ini, but I rarely encounter those anymore. It used to be that every ZIP had a bunch of thumbs.db files but Microsoft seems to have cut that out.

Alternatively, https://www.google.com/search?q=intitle%3A%22index+of+%2F%22...: "how to tell if an open directory index was created by a macOS user"

Not just zip files, USB drives too.

Came here to say this. Leaving satisfied.

Also, how to tell the new Sr dev mgmt just hired is really a jr dev.

Yes, I am talking about .gitignore

I can’t tell if you mean they included `.DS_Store` in their ignore file or left it out, but including it is just good practice.

Some past related threads:

.DS_Store - https://news.ycombinator.com/item?id=26435783 - March 2021 (30 comments)

Ask HN: What are the technical justifications for keeping .DS_Store? - https://news.ycombinator.com/item?id=23648165 - June 2020 (91 comments)

.DS_Store - https://news.ycombinator.com/item?id=20340151 - July 2019 (21 comments)

Don't commit your .DS_Store files - https://news.ycombinator.com/item?id=17134324 - May 2018 (2 comments)

.DS_Store files always hidden from Finder in macOS Sierra Beta - https://news.ycombinator.com/item?id=12056993 - July 2016 (11 comments)

.DS_Store for non-mac users - https://news.ycombinator.com/item?id=5733022 - May 2013 (15 comments)

Death to .DS_Store - https://news.ycombinator.com/item?id=3390509 - Dec 2011 (122 comments)

I'm curious, do you keep a list of old threads for popular topics manually, or do you just use the search function on the fly?

...Or does he just remember all past HN threads, and can easily out their entire titles, URLs, dates, and number of posting from memory on the fly, writes poetry about his cat, and is lots of fun at parties, like Data?

https://www.youtube.com/watch?v=DQCjqS3k2To

There used to be someone on Reddit who knew every scene and episode of The Simpsons and would respond to every comment he was called into to regurgitate his knowledge, so while your comment might be a bit over the top, I'm sure it's not super far from the truth!

@dang, if you see this: Can you fix this for me? It's hard to be a part of the discussion here with this type of thing: "You're posting too fast. Please slow down. Thanks." (I can reply to email sent to the one in my profile for verification, I can't seem to originate emails from there, though, as it's a Fastmail "masked email".)

I use HN Search on the fly but have a bunch of keyboard shortcuts in a browser extension to make it faster. Past explanations here if anyone cares: https://news.ycombinator.com/item?id=28441182.

Since Windows and MacOS don't seem interested in having mutually supported advanced filesystems, it would be nice if we just standardized on a flat file structure to implement portable features.

fat32 is very portable but has a 4GB limit and lacks entered attributes. exfat probably supports everything but until very recent open source drivers it lacked cross platform compatibility.

Plus too many devices are going to be limited to fat32 so we might as well just use that as a common denominator.

It would be nice if a DS_Store system could be used to associate split files (e.g. $NAME.1 $NAME.2, etc.) as a word around for the 4GB limit. User permission attributes and other extended attributes could be stored as well.

What's interesting is that Fat32 had this feature implemented on OS/2 and Win NT via files with the "␠EA.␠SF" suffix. Unfortunately looks like it was dropped in Windows 2000.

exFAT seems to be a decent compromise.

exFAT is great other than its lack of journaling, which has bit me in the ass more than a few times. file system durability is really important, especially for one for a drive being frequently moved between environments

exFAT works pretty well. In the past I have used UDF (yes, usually for DVDs) in external hard disks until exFAT support became common.

There are some licensing considerations with exFAT.

Not since 2019. https://www.phoronix.com/scan.php?page=news_item&px=Microsof...

This is also when exFAT support was added to the Linux kernel.

I’m not sure this resolves anything. The disk format is no longer secret, true, but the patents are still in place and I don’t see a license anywhere in the link. (Even the one that’s present in the FAT spec released with UEFI is only valid as long as you are using it to implement UEFI.) Linux specifically gets a free pass because Microsoft joined the corresponding patent cartel (the “Open Invention Network”) in ’18, but FOSS at large is still at risk.

(I also don’t see why anyone would implement the allocation-bitmap-based exFAT instead of the extent-based UDF. They haven’t even added a third FAT! ... But that’s an unrelated matter.)

So, why aren't these just stored in a db or some other directory of the OS? Instead of littering the userspace, and no fs-ext required. Also it sounds like these should be per-user-and-dir instead of just per-directory.

So that when you copy/compress/make disk image of/whatever a folder that you have customized (e.g. marked files as red or whatever or moved them in specific locations) they are as you set them when you or someone else opens them in another Mac.

Initially, because the early Mac OS X Finder was a bad enough set of compromises and slow as molasses without adding database lookups to another part of the file system (at a time when HDDs were universally spinning rust and iBooks and PowerBooks often trucked along with 4200rpm drives). The .DS_Store files could be there already in the directory and hidden from users with a flag while Apple figured out whether UFS or HFS+ was their future file system.

Now? Well I/O performance is hardly a consideration for the latest Macs, but they’re there and they work, except the Mac OS X Finder 20 years later while far better than it’s debut performance is still a load of crap so it can still forget your custom view settings for a directory.

they are only created on filesystems the OS believes cannot support metadata like you suggest

That wouldn't be a relevant condition if it always tried to use the user homedir or equivalent.

Ah, .DS_Store. As a Windows user forced to endure this offensive clutter I curse the fucking bastard that came up with that.

No love either for the twerp who invented desktop.ini or caused thumbs.db to seemingly keep coming back after I repeatedly turn off that feature.

If you're interested in such voodoo arts check out NTFS Alternate File Streams too (which while the bane of many at least have the decency not to spread to filesystems that don't support them).

This wasn't much of an explainer, it didn't even cover all the things that .DS_Store is capable of managing. The wiki does a better job of covering the topic.

It's also not something to get upset about - if .DS_Store or Apple double files (or for that matter Desktop.ini / thumbs.db) are a surprise then it's time to hang up one's coat, they're trivially revealed in the terminal and so predictable and easy to deal with that complaining about them is akin to holding up a dunce sign.

If one has purchased hardware/software that can't cope with them: get a refund, this shit is entry level basic.

Which wiki?

Sorry for being vague I mean this wiki: https://en.wikipedia.org/wiki/.DS_Store

Finally I understand the perspective of mac users in this problem. I had assumed that showing hidden files and manually cleaning up the zip archive would be a "simple no-brainer solution".

I just can't comprehend the reasoning of hiding .DS_Store beyond the reach of viewing hidden files. This is user-hostile. Why don't the native zipping tools automatically ignore these files when creating an archive?

So is there a way to get my Mac to stop making these? Dragging file icons to arbitrary positions in a directory is not a feature I want.

No because they include a lot more information than that. For example, where files in the recycle bin came from is stored in DS_Store.

I know this because there's a bug in MacOS they refuse to fix where you can't restore files put into the recycle bin programmatically. This is because the API doesn't write the correct stuff to DS_Store.

Further, the format is proprietary, and I've even read that its structure changes depending on how it's used.

Frustrating crap.

> where files in the recycle bin came from

Great, another feature I don’t want.

TLDR: .DS_Store files are Desktop Services Stores, containing that folder’s custom attributes, things like icon positions, and in more recent versions of macOS custom settings for the display of file metadata. Among the most important for some users are Finder or Spotlight Comments, which are normally displayed in the Comments section of the Get Info dialog for a file.

How do other OSes handle the same functionality without placing machine files in the users filesystem?

On filesystems that support the features: in the file system itself. On filesystems that don't support the features: somewhere on the system itself (like in the registry on Windows) or not at all (for example, when trying to set permissions on FAT32). Other operating systems generally don't create hidden files for this reason.

They do, however, create their own versions of the recycle bin/trash folder. If you use a flash drive on macOS, Windows and some Linux environments, you'll end up with three different recycle bin/trash folders. A bit annoying, but those folders are usually empty anyway.

Windows warns about the inability to copy permissions to FAT32 (in some cases, at least) and some command line tools on Linux do as well, but it depends on the tool.

It would be nice if Linux at least could see the Recycle or Trash folder and use what’s already there. Even better if both macos and Windows started doing the same.

My reflex of rsync -avh -P /foo /mnt/exfat-partition will fail in a confusing way until you figure out how to copy everything but permissions.

Windows has the desktop.ini file to support some similar functionalities.

I was curious as well. Apparently in Windows, they're stored in the registry: https://superuser.com/questions/625854/where-does-windows-st...

By using features of the file system for storing metadata. Alternate file streams, journals, Master File Table, etc.

From my knowledge .DS_Store is something everyone hates and no one actually benefits from, I personally hate it with passion and there's several times I wanted to work at Apple to remove it. Is there a reason why macOS still keeps it, do they actually think it's a good idea, or they don't like to listen to users, or they didn't get to do it yet?

> from my knowledge .DS_Store is something everyone hates and no one actually benefits from,

Small correction then; "something non-mac users hate and non-mac users don't benefit from"

I doubt you'll ever hear a Mac user advocate for DS_Store files because by design they do their job best when you never realize they exist. I think for __some__ users once they realize icon positions are maintained there, they'll be a bit more defensive. For many users (myself included), since it can also control view settings for a given folder [0], that I would be pretty peeved to lose.

Computers create junk files all the time and carry a lot of bloat, and furthermore one person's bloat is another person's workflow.

No joke, back when I was still doing general helpdesk support we had some "sort of clever" users who found out that they could recover some project files from %appdata% and relied on this for "auto-saving" their projects. When a new fileserver was spun up and this time the admin forgot (decided not to?) to toggle redirection for Appdata, this user was in for a shock when we had no backup of their Appdata and they had lost months of work.

Personally DS_Store is a very minor annoyance at best, if even noticeable. For version control it's just another file to add to ignore lists, and for network shares it's a very simple MacOS command to disable writing them to network shares. But it's not fair to call it worthless, it has a lot of good uses if you're a Mac user.

[0] https://www.arno.org/on-the-origins-of-ds-store

.DS_Store files are also a PITA for Mac users who mainly work on the command line. I don't understand why this data isn't stored in filesystem attributes instead of 'user-visible' files (for some reason I didn't have problem with .DS_Store on my previous Mac, but now on the new Mac they're suddenly everywhere again, might be that I somehow disabled them years ago on the old Mac though).

There is an awesome Atom plugin called "ds-store-delete" which allows me to use ctrl-alt-x or click a bright red button in the bottom left corner to delete all .DS_Store files in a workspace.

https://github.com/pedroparra/ds-store-delete

Adding a bright red button for something so unimportant? .DS_Store should be in your .gitignore anyways.

I don't even want to see the .DS_Store files at all :) (I am aware of the checkbox to hide gitignored files, but still like to see everything)

I just realized that once my girlfriend (against all my advice...) gets her new Mac, these fuckers will show up on our NFS server, huh?

Does anyone know how to prevent that from happening? It's a non-auth NFS, so anyone on the network is supposed to see and use it. So a server-side solution would be preferable, but client-side will do in a pinch.

Client side: https://osxdaily.com/2010/02/03/how-to-prevent-ds_store-file...

There’s always been a hidden setting to stop Finder from generating these on a server, but the downside is that it won’t remember things like your girlfriend’s view preferences on the server (window position, sort options, text size, view type e.g. list or icon view).

That setting is also per user account, so if you have a user account for yourself on the same computer you would have to set it for yourself as well.

Samba has a setting to forbid creation of that shit, I don't know about NFS though.

iptables can forbid her to access the NFS. I have a script to delete them with cron scanning the entire drive, since I have the same weakness for Mac users as you.

It makes sense to hide these files from many macOS users who don't care about them. For developers (who should be able to work with a Terminal) there's no need to resort to any magic or special command to make them visible! They are normal hidden files. A simple `ls -a` will list them too. You also have to put them in .gitignore as you put desktop.ini and other window managers' metadata files.

I wish the author explained in more technical detail or clearer how and where these files cause problems (not doubting that it does). In that case, it would have made a really useful article.

Fyi, if you use netatalk to expose network file systems to macOS, you can tell it to store these things as ext4 extended attributes instead.

Samba has that option too, but I prefer not to store those things anywhere.

The Veto Files option of Samba springs to mind

How do these work with multiple users? If two users are looking at the same directory, then do they fight?

More than the DS_Store files what fascinates me is that when you delete something on a USB stick it doesn't go away, making the life of the stick limited to one write. This, unless you format it, but how many regular users understand it? The champions in usability...

Come on…an "explainer" of .DS_Store that doesn't even mention resource forks is misinformation.

It just used different words. "Sometimes they’re confused with another annoying but more useful hidden file: shadow files whose names start with ._ which are used to carry extended attribute data as part of the AppleDouble file format used on some FAT file systems."

Yep, .DS_Store files have nothing to do with resource forks.

Resource forks are a filesystem feature (well forks are a filesystem feature and resource forks are an application of that feature). AppleDouble is a way of storing the information that would be in resource forks on filesystems that don't provide forks or similar functionality.

They have everything to do with resource forks. The metadata in DS_Store is stuff that was originally stored in the resource fork on earlier MacOSii; the file's existence is a cludge created to deal with the lack of resource forks when the decision was made to design OS X with POSIX-compatibility.

No.[1][2]

[1] https://en.wikipedia.org/wiki/.DS_Store

[2] https://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_fo...

Fond memories of messing around with ResEdit in classic MacOS intensify

Reply on news.ycombinator.com