Parsing the .DS_Store File Format (2018)
0day.work
0day.work
Anyone have an idea why modern versions of MacOS still uses a .DS_Store file? The OS could just as easily read and display the contents of a directory. And deleting a .DS_Store file doesn't seem to have any noticeable negative affect.
Basically it seems to be the replacement for the old resource fork data on non-Apple file systems.
Custom icons for folders are stored in a special "Icon?" hidden file, but not actually in the file's normal data-- they're in an extended attribute too.
(The xattr in question is called "com.apple.ResourceFork" and I'm presuming is structurally similar to a resource fork in classic Mac OS.)
One thing I've wondered is if this method still makes sense on modern hardware compared to keeping all that information in a centralized sqlite file or something along those lines. I could see individual files being advantageous on a slow mechanical HD where seeking through a file that could eventually grow large might pose issues, but that shouldn't be a problem on an SSD.
The main advantage I can think of that might still be relevant today is that it makes Finder window state machine-independent, so when you open a folder on two different machines it'll still open in list mode sorted by Date Modified or whatever.
Not sure if it's "Desktop.ini" I'm thinking about, but here's some documentation on it: https://learn.microsoft.com/en-us/windows/win32/shell/how-to...
The Wikipedia article is a pretty good resource for this: https://en.wikipedia.org/wiki/.DS_Store
(Dotfiles are per-user system configuration options written as a file in the user’s home directory. This means that for any system on which that home directory exists, that user’s configuration options will be applied. If the home directory is shared across multiple computers over a network, or if it is backed up from one computer and restored to another, then that user’s settings will persist.)
If these settings were stored in some other central database then if you copied the macOS folder from one computer to another, your copy tool would have to know how copy the settings as well. Keeping those settings inside the folder itself means that your copy tool only needs to know about copying folders, instead of copying folders and also copying settings.
It’s one of those Unix principles where everything is treated as a file: an elegant technique for a more civilized age.
MacOS prior to X had much more elegant solutions to the problem. To this day, when you switch to/from icon/list view in the Finder, more windows than just the current one switch. In MacOS 9 that loosey-goosey unpredictable behavior never happened.
And there's almost no way to make this behavior stick because... it's different if you click through folders and if you click on pinned folders in Finder sidebar.
I mean... How?!
For the exact same reason Windows has a Desktop.ini file?
They store per directory settings for the Mac Finder and Windows Explorer.
Storing a local file inside the directory allows for moving that directory around while also keeping those settings.
The file is invisible for users of Finder so it is still the best solution for this task.
I have a .DS_Store line in a global .gitignore and don't notice it much but when I'm annoyed by DS_Store files in a specific directory, I usually bulk delete them using `fd`:
fd -u --type file --fixed-strings .DS_Store $HOME/Projects/some-file-server-project -x echo rm {}People working on Linux or Windows do not need to know about garbage your dev env leaves on your machine.
I’m saying this half-seriously as I fully understand that these are so common, it’s more convenient to have the exclusion synced between clones.
That said, as a purist, none of the repos I’m watching over have .DS_Store in their gitignore
I understand the pollution concern… but unless that .git/info/exclude is standard in the company, what prevents a intern to not have a correct exclusion there, and happily push a .ds file ?
.gitignore looks more robust to the obvious user. And it’s not like it’s a file you need to look at attentively every day.
But yes, I do see that you are correct!
Our CI.
And code review is what prevents .DS_Store to be added to .gitignore.
And, again, I'm saying this as a purist with a developer team of 99% Mac users.
Just like we have CI checks to make sure nobody accidentally commits a secret (like the GitHub host key thing last week) we have checks that prevent debris to be committed and code to be formatted according to agreed-upon coding standards.
All of this might seem superfluous when the live expectancy of a repo is measured in months or single digit years, but then, no solution or repo is more permanent than a quick throw-away one.
Which is why this isn't even a discussion but just a reality. Been there, done that, learned my learnings.
Instead, you appear to be enforcing in which arbitrary location to place ignore rules for debris, seemingly having spent considerable time implementing that, for entirely puritanical reasons.
And while you're free to play holier-than-thou at your job as you like, I'd give hell to anyone wasting my team's time like that.
As a DevOps guy (but with 30 years of experience in development too), one of my pet peeves is having to deal with a thousand different edge cases because someone decided that intellectual cleverness was more important than a holistic approach to design and architecture.
They're "puritanical reasons" right until you have to migrate to a new version control system, at which point they're the rules that all migration tools enforce -- because those are usually written against the two VCS' specs and current implementations, rather than the myriad of alternate practices that software shops everywhere devise.
I've done migrations like these before -- while the parent poster may be doing all that for the wrong reasons (puritanism) they are absolutely right to do them.
The "myriad of alternate practices" that you mention are a strawman: We're still talking about ignoring some file manager metadata file. Developers put a line into their .gitignore and be done with it. If that breaks your new VCS, maybe migrating to it isn't such a good idea?
Not some arbitrary files, but the ones that the VCS recommends, or at least allows, to be excluded locally, where it makes sense. See e.g. github' note: https://docs.github.com/en/get-started/getting-started-with-... on the matter.
Also not a hypothetical new VCS, I've seen this backfiring several times, and it's usually not related to how good the new VCS is. E.g. when doing a ClearCase -> SVN migration years ago, just cutting out a few global exclusion rules that should have been local (e.g. tags files for people who liked etags, cscope.files for people who liked cscope) reduced a full-history migration time from several days to a few hours.
The two tools did not treat filename encoding the same way, so the migration tool had to walk through the exclusion list at every history point it synchronized. Due to how ClearCase presented its repositories (tl;dr userspace filesystem, years before FUSE) this was very slow on its first Linux versions (it had originally been offered for commercial Unices), not so much because ClearCase sucked but because it exposed a nasty quirk in the kernel's VFS implementation.
> The "myriad of alternate practices" that you mention are a strawman: We're still talking about ignoring some file manager metadata file. Developers put a line into their .gitignore and be done with it.
We're talking about ignoring file manager metadata files specific to some developers on some machines. .gitignore is global. Some details matter.
If you don't expect to ever do cross-platform migrations -- of even much cross-platform development, for that matter -- for repositories maintained across multiple platforms, yes, sure, you don't need that. That doesn't mean OP has a twisted view of adding value. Maybe his team does need that.
Edit: FWIW, migrations are just the nasty point that bites you back years later and usually comes with a huge bill. But puting all local exclusion rules in a single global file backfires in all sorts of ways on large and/or long-lived codebases. Sooner or later some eclusion rule specific to one developer's environment will do the wrong thing in another developer's environment.
I've seen all kinds of filters in a .gitignore for programs I don't personally use. I don't mind it at all.
I'm pretty sure I'll never create so many new projects in my entire career that those 8 seconds would add up to the amount of time required to follow your suggestion. Be a little more pragmatic, folks!
Using `~/.config/git/ignore` requires no extra setting.
> Patterns which should be version-controlled and distributed to other repositories via clone (i.e., files that all developers will want to ignore) should go into a .gitignore file.
and
> Patterns which are specific to a particular repository but which do not need to be shared with other related repositories (e.g., auxiliary files that live inside the repository but are specific to one user’s workflow) should go into the $GIT_DIR/info/exclude file.
To me it seems obvious that all developers would want to ignore .DS_Store even if only some of them would generate it. And certainly you want it distributed to other mac users when they clone. Further, it's certainly not the case that .DS_Store is specific to just one user's workflow. On all three cases therefore gitignore would be reasonably appropriate.
I stole this from somewhere: Create a .gitignore_global file and include .DS_Store, then in the [core] of your .gitconfig include excludesfile = /Users/janedoe/.gitignore_global.
git config --global core.excludesfile ~/.gitignore-global echo .DS_Store >> ~/.gitignore-global
I hated it, most of people hated it. MS finally got rid of it (in Vista or 7) and store thumbnails in a centralized place.
However, I now start to miss it.
I store lots of large image files (think 1200 dpi PNG scans) on my computer, and it took Windows years to create thumbnail for them, while locking the entire hard disk (and sometimes the whole explorer.exe) in the meantime. This process, while obviously is much slower on spinning HDD, it surprisingly isn't really that fast on NVME SSD, either.
This would be fine if it's a one-time process, but unfortunately the aforementioned centralized thumbnail cache either have an expiration time, or have a size limit (or both), so the reality is if I didn't visit a folder for awhile, its thumbnails are all gone, so they have to be generated again, make the explorer unresponsive for some time again.
I have been annoyed by this for long time but I don't have any solution. The best workaround is to only view such folders in list/detail mode, and rely on 3rd party image manager (I use XnView) to show the preview/thumbnail -- which has a much faster thumbnailing speed and more robust thumbnail cache management.
Windows is renowned for having awful performance in usecases involving accessing many small files even for read-only applications. I know a team that even went as far as testing switching away from Windows to build Windows apps and instead switched to cross-builds on Linux just to avoid that performance sinkhole. Meanwhile I worked on a multiplatform project where macOS builds on a Mac mini for the exact same project took less than half the time to build than on a beefed up Windows workstation.
Here's an interesting discussion on HN from a couple of years ago on this topic.
More info here: https://superuser.com/a/298798
My question is: Linux and Mac both have the ability to store key-value pairs on files/folders with xattr. Why did or does Apple continue to use .DS_Store?
Possibly because not all copy programs copy metadata.
Possibly it's just technical debt: .DS_Store was added to MacOS before xattr and rewriting the Finder to use xattr is not high priority.
But at the very least, they could set the FAT hidden attribute on this stuff. It must be some mentality like "Well, I'm not littering in my own yard?"
Personally, having done so recently to reverse a custom game archive format, I have only used the usual hexdump, strings, grep, sed, etc. I'm wondering if there are more powerful tools to do this kind of job?
- 010 Editor (commercial)
- Kaitai Struct (free & open source)
Also has the advantage of directly generating parsers from the defined structures.
http://fileformats.archiveteam.org/wiki/Desktop_Services_Sto...
I'm on Big Sur (11.7.1) and I didn't find a way to make finder NOT create them.
It's (one of the many) irritating idiosyncrasies of this OS
You can however create a cronjob that deletes the files as often as you want, the operation takes 11 seconds on my 1TB SSD with a fairly large home folder:
fd -u --type file --fixed-strings .DS_Store $HOME -x echo rm {}
I've always hated this excuse from software developers (I know it's not your excuse, alin23).
User: How do I stop my computer from doing Unwanted Thing X?
Dev: You can't. Unwanted Thing X is needed so we can do Unwanted Thing Y.
Fucking A! Stop doing unwanted things on my computer! My computer should only be creating files I command it to create. It shouldn't be doing things I don't want it to do simply because it conveniences a developer living 1000 miles away in Silicon Valley. My filesystem is there to store my files not some developer's metadata. It should not contain this trash from my operating system. At least give me a DONT_CREATE_THIS_DS_STORE_CRAP environment variable or System Preference to set!
I'll be generous and grant one root directory tree to the OS for its litter. Linux can have /var, macOS can have /System and maybe /Library too. Windows can have C:/WINDOWS. Everywhere else in my OS, hands off, please!
I guess you could block .* except .well-known 'just in case'