Ask HN: What are the technical justifications for keeping .DS_Store? | Hacker News Reader

Ask HN: What are the technical justifications for keeping .DS_Store?

106 pointskaptain91 comments

I was hoping that with macos 11 we would see the deprecation of .DS_Store. I'm not running the beta so perhaps it is, but what has surprised me is its sustained longevity. What are the technical reasons for keeping .DS_Store? It is essentially a application-specific (i.e. Mac Finder) solution that pollutes the data store.

Off the top of my head I can think of a number of other solutions to store directory metadata: a Finder-specific database mapping values to directory paths, resource forks (ala OS 9), etc.

View on HN

I'm going to be honest: it should be deleted when seen, and OSX should have never begun this.

99% of the reason it exists is because the OS generated thumbnails. I see these in, for example, zips, that do not have images in them, and have that file.

You know what I've seen? People do `git add ` and then I find that directory in the repo. Why does git not automatically ignore that?! It's never the right option.

Anything* that pollutes directories should be absolutely verboten, its never what the user wants.

Edit: The worst part is, it makes them on network shares, and on filesystems that aren't HFS. The hell, Apple?!

> 99% of the reason it exists is because the OS generated thumbnails. I see these in, for example, zips, that do not have images in them, and have that file.

QuickLook thumbnails actually go in a different cache as they are dynamically generated. .DS_Store files contain things like folder layout information.

> Why does git not automatically ignore that?!

Why doesn't Git automatically ignore desktop.ini? .desktop? __pycache__? In my experience, people end up creating bespoke .gitignore files for each project and sometimes shared online, or systemwide. I don't know if Git should make it global.

If Git started not committing some files in a directory without user input, based on conventions of an OS or programming language that you might not be using, that would seem like quite the ad-hoc side-effect.

I would definitely vote for a way to incorporate sane .gitignore settings per-OS or per-language, but still make it opt-in.

Just add it to `~/.gitignore` and be done with it. Yeah haven’t worried about this in 10 years.

Or even better, XCode, Brew, and other services that have git can set it globally when git is installed. They know about MacOS after all

Adding to `~/.gitignore` is not enough. You need to tell Git to use it via `git config --global core.excludesfile ~/.gitignore` or similar.

That approach works for one person or like minded dev teams perhaps.

it fails in many teams, they do not care or a junior dev does not know. Many folks don't setup proper dotenv especially designers. Unless added to the project .gitignore file it is going to be added to the project sooner or later.

> it fails in many teams, they do not care or a junior dev does not know

If your team doesn’t care, by definition they don’t see it as a problem, so it only doesn’t work in the sense that they have decided hat they don’t need a solution.

If your team cares but your junior devs don't know, you need to work on your standard practices documentation and onboarding procedures. “Use this default baseline .gitignore" is not a very arduous level of standardization that is hard to instill in new juniors.

It is easy to explain and help someone with this. :)

Is it just me or nobody actually have problems working with non technical folks on git ?

Getting a team of designers, content writers, marketing folks to use anything beyond their daily tools over a period of time is hard.

I had the marketing team run a git managed, asset pipeline ,CI deployed Hugo static site via a CDN for three years, with great performance and high availability for three years, now we run hubspot.

Once the initial team moved on, It was next to impossible to hire people willing to work on that stack , typically new guys will know Wordpress or hubspot etc, my TAs had frustrating time sourcing, I finally had to give up .

Administrative Config in yaml/JSON files in a git repo sounded a nice idea , git gave version history rollback, audit log, branch/env, CI driven deployment all sounded great.

Same problem typical operation folks are not used to this , they want a nice UI , and keep clicking and rather than do a grep/Sed replace in a jiffy.

People just like different ways of working , it is hard for me to think graphically like a designer , I can understand that it can be hard for someone to think in text , it is better we give them tools that are comfortable with and fight other battles

Genius. Never thought about doing this.

I much prefer it to be opt-in rather than opt-out. For any git project I start, I always visit https://gitignore.io.

zsh has `gi` command which you can do like this

gi macos node > .gitignore

Is it not normal to have a repo or something somewhere just for your starting "boilerplate"?

Whenever I switch computers, one of the very first things I do is copy over my bindings in CAD so that I don't have to relearn / figure them out again. I don't see why it'd be so hard to just store your own somewhere, and use it over and over again.

- I used to setup zsh Git aliases based on zprezto but with some of my own tacked onto it. But you can't install zprezto on Windows Git (which uses bash, not zsh). And I switched to fish on Linux, so I lost my aliases there too.

- Different programming languages have different files to ignore. I hop between many languages.

- It might be a bit weird for multiple repos to have the same root commits. Not sure.

git can be configured to ignore .DS_Store by default - in your home directory, make a .gitignore file using the usual format, then do "git config --global core.excludesfile ~/.gitignore".

Better still is to put it in ~/.config/git/ignore and leave your core.excludesFile unset, as that's the default path and will therefore live right next to your config file.

all your team members have to also to do this, some of them like designers are not always comfortable in the command line

~/.config/git/ignore is a text file, not command line

By default “.” Folders are hidden in Finder , either they need reconfigure their Finder settings , or just copy paste a echo pipe in terminal. Both options are cumbersome to get others to do.

They also have to redo everytime they reset their system. Since Mac will not show the folder in Finder they won’t even notice when they don’t do it , it is annoying for the rest of us.

In windows and Linux you can still open hidden folders and files simply by typing the hidden path element into the file manager. You still won't see any hidden contents, but you don't need to type out the full path--just navigate to your user folder, and type `./.config` in it to enter it.

The whole problem is with Mac adding arbitrary contents to folders which may be used by other operating systems!

Like the MS of the past, Apple does not care about compatibility unless they forced to.

If an individual is unable to figure out how to show dotfiles in Finder on MacOS do you really want them making commits?

Well it is preferable than getting a bunch of emails with filenames like (Copy)(Copy)_final_(11)__27th June.

Git should not be limited to few folks who understand version control, there is a lot value for people who can just it linearly for collaborative history tracking

It sounds like you should just stop being bad at teaching your team members.

Thumbnails were a mistake. thumbs.db is just as dumb as .DS_Store, I don't know how anyone thought this was a reasonable solution.

I've posted reasons in post over here, do you find them baseless?

https://mike.place/2020/global-gitignore/

just leaving this tip here, if you use node.js, `npx gitignore node` generates the appropriate git ignore files for common ignored files including ds store

.DS_STORE is the only entry in my global. gitignore.

.DS_Store is like the filesystem equivalent of 'Sent from an iPhone' --- it lets everyone know you used a Mac.

I've always suspected this and the rest of the conversation confirms it.

Personally I hate .DS_Store and they must be deleted. BUT on the other hand I am finishing "a proper missing, a dream File Manager" for Mac OS and thinking about features that would require to store some meta data. Where should I store them? If it's in the folder in form of some ".meta-data" then it would be copied when you copy data and be available there in another machine, or I can create it inside the OS somewhere, but then how to transfer it to usb drive for instance? I do not like idea of data to be polluted with some extra .something, I really hate the idea, but how to deal with it to keep it manageable? I can make some indexes/pre-calculated sizes in the root of the drive, if user chooses to have those, BUT then if we speak about shared network folder, it then may very well appear in some folder after all because it was mounted as a drive, so the only solution would be to put it in ... well the same ".meta-data" just like .DS_Store that I hate. Then only thing to do about it is to give a choice to user when he copy something, to avoid copying those too. Right now I am using constantly "my file manger of a dream" and actually almost forgot about .DS_Store disaster, because honestly I have no need for poor Finder anymore, and thus .DS_Store is not created at all, but should I add something like this for extra features which would -really- require them? What do you think? Putting yourself in the shoes of someone designing File Manger you kind of see where .DS_Store comes from. Still, hate it :) Really thinking what to do for some time now ...

Store your app's configuration, including per-folder view options, in a centralized store -- either ~/.dreamfm/ or ~/Library/Application Support/ Dream File Manager/

In the same view options configuration window, have a checkbox option to make the configuration portable along with the files, in which case write it out to .DS_Store or a similar file, to be used as a local override à la .htaccess

Explaining this option succinctly may be a challenge but not impossible. Maybe "Save view options with folder" with a hover tooltip explaining that this would enable the view options to remain the same across user accounts and computers.

Thank your for your thoughts. This is more or less how I see it too in general.

> Maybe "Save view options with folder"

I like it, it's not only 'view options' though, as you understand. Many '.DS_Store' files are already there for many folders and I have to deal with them while compare sizes etc. Right now they are copied because I do not think FM should be invasive in your data as a general principle. But some option along the lines of "Copy meta data too" I think is reasonable way to deal with them.

If I'm moving files to a USB, it may also be to move things not just to a different machine, but a different OS. Perhaps even a different file manager, so your file would be the same pollutant. So overall anything like .DS_Store is unwanted.

As mentioned by other comments, store it in a centralised cache/config/whatever for your application.

I agree, I do not like pollution traveling around data. On the other hand I think FM shouldn't be invasive if this pollution is already there. Option can be added to deal with them, but only if you choose to, because not always you wish to. Many times you just wish to copy it 'the way it is' without a headache and even this simple basic task is not always possible to achieve with Finder.

Compute a simple checksum over first megabyte or so of the image file (for O(1) complexity) and store it per known file in memory. Store thumbnails using that checksum as key in global cache. Don't purge because methods discussed in this thread don't.

So, to show thumbnails for a directory with 100 photos, you’re going to open and close 100 files, and read 100 1MB chunks of data?

That probably is doable on modern local hardware, but on network disks, it might still be slow.

Make that an IMHO not unreasonably large 500 photos and your local disk might be slow, too.

Add a virus scanner, and 100 photos might already be annoyingly slow.

⇒ I would try to find a way to store away the checksum somewhere in the file (say in an extended attribute, as a (modification timestamp, checksum) pair)

Normally to show thumbnails you need to parse the whole image anyway. Unless that is the format doesn't have downscaled data already (progressive jpeg/PNG)

If you don’t cache thumbnails, that’s true, but that’s too slow, so you cache thumbnails.

If you do that, you only have to read the whole image the first time. Once you created the thumbnails, you only need to read them, and they’re a lot smaller, and can be put in a single file.

The proposed approach reads a significant part of each file to determine whether the cached thumbnail is still current. That doesn’t give a huge improvement over not caching thumbnails at all (factor of 5 in bytes read/network traffic, as a rough guess, likely less in disk seeks)

It's a dilemma of memory vs computation. It depends on what we expect/prefer:

FM that takes a lot of memory or FM that use computation resources when required only. I tend to think that in general expectation from FM is to be very small in memory demands and take valuable computational resources only if it is absolutely necessary.

My approach would work amazingly in ipfs where hash of contents is an intrinsic property of the file system

You're right. I got a bit ahead of myself it seems.

Thank you for thoughts though. Was interesting to read both of you.

> I am finishing "a proper missing, a dream File Manager" for Mac OS

Can I sign up to be notified when it's released? :)

Sure :) I understand you . For me Mac became unusable on a new setup until I install my FM. I wrote it for myself in the first place and I am using it heavily daily. Working on it for quite some time now. I do not plan it 'small' as another BS app over there. This one should really 'just work' in the best mac traditions of SJ we know: elegant for beginner and powerful for advanced in the same time, with high quality standard in mind as the first priority. So consider yourself being signed up for the first notifications, you won't miss the 'revolution' ;) Hacker News were very helpful and supportive resource for me and I think the first public announcements are deserve to be here.

.DS_Store is the new Thumbs.db. Our descendants will keep finding them in the backups and scratch their heads.

MacOS does have the ability to store any arbitrary metadata in extended attributes, which would indeed seem to be a far better solution here.

It doesn’t really bother me as it is, but it’s on the less-than-perfect list.

Extended attributes are lost by a lot of mechanisms of moving files and folders around (e.g. zipping them with something other than Finder's "Compress" command). It's also not portable to other filesystems, whereas a .DS_Store file will work on any filesystem.

that seems fine?

Losing metadata by accident seems fine?

.DS_Store is not metadata. It is configuration information for Finder. It is not necessary for each computer looking at a set of files to use the same, e.g., icon size.

And if it were a real metadata. Would you then see justified to have some .meta-data file with data?

That battle was lost long ago and the answer is generally "no." Classic MacOS used Resource Forks. That's where file type was stored instead of using extensions. I believe icons were stored there, as well as other things like where you downloaded it from (a favorite feature of mine in Classic MacOS). Zip files would write that data a special folder. Likely, because this was so problematic in a multi-OS, Internet-enabled world it was ditched for file extensions and encoding everything else inside the file. If you downloaded a file from the Internet that wasn't kept in an archive it would be useless without metadata.

Today, macOS uses extended attributes (which were already mentioned) which stashes things like where you downloaded it from, if it's quarantine (first launch warnings), if it's a screen shot, etc. Funny enough, when you place something in the Trash it stores the former location (so you can put it back) in DS_Store.

Generally, metadata is either stripped or just plain gets messed up when switching OSes depending on the partition type and tools used to transfer.

“Classic MacOS used Resource Forks. That's where file type was stored instead of using extensions.”

That’s incorrect. File type and creator were stored in the file system, not inside files.

“I believe icons were stored there”

Icons typically were stored in resource forks, but not in the way I think you imply. Applications defined mappings from (type,creator) to icon in their resource forks (which also stored lots of other stuff, including application code). The Finder merged all the mappings needed in the desktop database (each disk had one, stored at the top level of the disk)

The icons couldn’t be read from each file because opening each file would slow down the Finder (it was slow enough to start with) and, I guess, it would take way too much disk space (a single 32×32 black and white icon with mask is a whopping 256 bytes. Overhead probably made that a kilobyte on disk. That adds up on a 400kB disk)

Thumbnails aren't metadata, they are a cache, since they can always be perfectly and deterministically recreated from the data.

Thumbnails aren’t stored in .DS_Store.

DS_Store is stored alongside the files, which means it's portable with them, e.g. when you put files on a USB stick and then plug it into a machine running an earlier version of macOS 10.

None of your solutions preserve that backwards compatibility -- the database wouldn't because it presumably stays on your machine, and going back to resource forks wouldn't because the earlier OS releases wouldn't know about the existence of them.

We have iCloud now and ETE encryption. The proportion of people plugging USB devices into machines they don't own is probably pretty minimal.

Still if you have pre calculated sizes of the folders, does it really matter what media was used to transfer data? Also I think people realising more and more that it'ss not very rational to trust to anyCloud with personal data. My rule is simply never use any cloud.

Windows has thumbs.db, but seems to do a reasonably good job not copying it into stupid places.

And Linux Dolphin creates .directory files. Fortunately, there's a configuration option to turn it off. These kind of files seem to be usual for file managers.

https://www.reddit.com/r/kde/comments/avmsdt/dolphin_is_grea...

Windows still creates thumbs.db into my network drives.

Synology pollutes my directory with @eaStore and @syno crap.

WHY?

I have Mac, Windows, and a Synology. I also sync this to Google Drive (using rclone) as a backup. My network shares, and Google Drive, are polluted with crap and I don't know how to fix it :(

WD MyCloud devices create a .wdmc directory in any folder that has images or videos. I've solved this issue at various times by adding a new file extension for mp4 files (mmp4, associate it with media player) until I changed apps to one that only displays video files; running a script after uploads to delete them; killing the service every time the NAS reboots (and I notice). Currently I'm using ftp instead of smb, as it doesn't seem to cause it and transfers faster as well--plus, ftp displays the hidden directories where smb doesn't.

    find -delete

Might be helpful.

I destroy them with reckless abandon and, knock on wood, haven’t had an issue in 15-ish years of driving OS X.

Doesn’t mean it’s right... but this is my anecdotal experience.

The Amiga had an equivalent: every visible icon was accompanied by a small metadata file appended with .info (More information here: http://krashan.ppa.pl/articles/amigaicons/). But it didn't 'pollute' the filesystem, and was far more manageable.

Haiku does things differently, and elegantly: https://medium.com/@probonopd/my-sixth-day-with-haiku-under-...)

Because the <name>.info was created by you (its major purpose was to have a custom icon to a program, drawer (directory) etc.).

If a file did not have corresponding .info file it wouldn't show up in the UI (unless you selected option to show all files).

I miss Amiga :)

Could always add this to your crontab

    */20 * * * * root find / -name ".DS_Store" -depth -exec rm {} \;

No need for "-exec rm", you can use "-delete".

And probably better to limit this to "~" rather than "/".

I disagree. If anything, inside your home directory is the one place where .DS_Store files are potentially useful and not harmful. And -delete has limitations regarding files that aren't within the current working directory or its subdirectories (I don't know what the "current working directory" is for cron off the top of my head).

Even if that's true, you should use xargs, not -exec

The Correct(tm) way: find [whatever] -print0 | xargs -0 rm

Well it's pretty simple. The Finder allows you to customize a folder (icon positions, background picture) and that data has to be stored somewhere.

You can tell the OS not to create them on network mounted drives:

  defaults write com.apple.desktopservices DSDontWriteNetworkStores true

> Off the top of my head I can think of a number of other solutions to store directory metadata: a Finder-specific database mapping values to directory paths, resource forks (ala OS 9), etc.

If you go for “resource forks”, you still would need a file.

And nitpicking, that wouldn’t be “ala OS 9”. That OS had a desktop database on each volume.

it would be nice if there were more settings for this

- external drives

- non hfs/apfs drives

- everywhere

Here's a brief and somewhat interesting explanation for why it exists, from Arno, the tech lead at the time (2006): https://www.arno.org/on-the-origins-of-ds-store

You could give https://asepsis.binaryage.com a shot—it's unmaintained and unsupported since El Cap, but it apparently still worked on El Cap if you turned SIP off. Maybe it still works.

Well, I could go on a rant around their purpose and some of us would ultimately decide these files like so many others are actually just debris.

Instead I will say we can have a set of critters that cull them in various interesting ways. Critters that watch filesystems for these files and remove them if they still exist an hour after creation. Or remove them all on a schedule. Or block list them via .gitignore and similar. Sync scripts that have block lists for files and folders that match names. The solutions are endless.

This isn’t only a mac thing either. Other OS create their own variations of debris.

It used to bother me, but not anymore. I always put it in my .gitignore files, and otherwise ignore it. You could alias “ls” to not show it.

I don't use macOS and even I cannot get away from them. Someone zips a folder and send it to me and when I unzip it, there would be a .DS_Store inside.

It's an approach that works reasonably well in the Unix world of OS X and platform agnostic servers. It plays nicely with non-Apple filesystems including network shares using non-Apple protocols. It also makes it fairly trivial to keep the metadata with the files when moving things around with generic Unix command line and archiving utilities.

I think the problem is Finder long ago move away from spacial interface to a browser interface. Since the move wasn't very clear-cut it's now just bad at remembering position and layout. Not to mention those things aren't very useful on a completely different computer that would have a different resolution and screen layout.

I think the data would be just fine stored as metadata on the directory or a centralized database that wasn't persistent between computers and the disruption would be minimal. Spotlight comments, which Finder reads from DS_Store, is also written to extended attributes. It just isn't read from there. Trash stores each file's previous folder in DS_Store, which could be problematic on a different computer (and nobody would expect that to persist).

If you’re enough of a power user to notice, then you should be able to figure out how to disable them.

A quick google search finds several blog posts with commands to run.

Did anyone create a bug report for Apple I can duplicate?

Apple usually prioritizes bugs by duplicate count.

These folders are really useful to identify immediately the clueless developers who put them inside zip files or commit them to git repos. They are a fast way to know that you do not need to take whatever these people do too seriously. I hope macos continues creating these stupid files forever.

"I hope operating systems keep putting up traps so that there's more ways to make mistakes"?

Yes, I agree that this is a bad idea overall. However, since I'm not a macos user I couldn't care less.

Reply on news.ycombinator.com