Should you remove duplicate files? | Hacker News Reader

Should you remove duplicate files?

eclecticlight.co

62 pointsingve61 comments

"Finding and removing duplicate files has long been one of the mainstays of Mac housekeeping."

What are Mac users doing that makes finding and removing duplicate files such a regular task? I don't use a Mac, but I can honestly say I've never given second thought to duplicate files. I'm sure there are some here and there but my hard drive isn't filling up with duplicates of large files.

Space is precious! Soldered SSDs can't be replaced, so we are stuck with the size we could afford at the time we bought our MacBooks.

If I had to guess, it might have to do with the Mac standard of “save on close window” which is a vestige of an old apple desktop metaphor where every window represents a file somewhere.

So you end up with a lot of duplicates from either trying to edit a read only document in some apps (which spawns an editable copy) or accidentally hitting duplicate (generate a copy of the current file in the same directory) which is the mac equivalent of “Save As”. It’s on you the user to move the new file where you want it.

There’s nothing special about Macs or Mac users, this article is just in a Mac-focused blog, so it’s not implying anything about other OSes. Deduping is a mainstay of computer housekeeping on any OS, for some people, though maybe not you. There are lots of workflows that end up with duplicates, but certainly not everyone uses them.

Here are just a few ways I’ve ended up with enough dupes to want to dedupe, which has happened on Windows and Linux for me as frequently as on Mac: Backing up mobile phones, using version control for multiple repositories that share files, losing access to repository servers and archiving or backup up the files, forking an old archived folder and making modifications to it that I want to keep, not to mention homegrown schemes & scripts for archiving and backing up when I either don’t want to pay for backup software or can’t even find anything appropriate to all my needs (cross platform, saving to a NAS, shared with my family, etc.)

Yeah, as someone who sometimes uses macOS, I was confused by this initially but I believe you have the right interpretation.

I do use a Mac. We spend a lot of time on maintenance. Here's my Saturday schedule:

* 7:00 AM find and remove duplicate files

* 8:00 AM reorganize the 2D layout of file icons within folders for maximum visual pleasure

* 9:00 degauss the LCD (really frustrated that you need third party software for this since the CRT days)

* 10:00 find and remove duplicate files

And so on. Finding duplicate files is just as essential as monitor degaussing, but you won't find this listed in the manual. You have to go to user forms where people will convince you of the need and get you to adopt procedures like the above to keep your machine at top performance.

Heheh, don’t forget to reset the SMC, it collects too much entropy overnight so if you want deterministic behaviour you’ll need to make sure to dump its contents to /dev/null.

You can also make your database persist writes to /dev/null, it will make performance go through the roof

Then you are going to need my app Dr. DevNull. It keeps your /dev/null clean and tuned for high performance writes. Only $5/mo. Upgrade to Dr. DevNull Pro to keep backups of everything written to /dev/null. Dr. DevNull Enterprise has all that plus SSO.

If you're not rebuilding the desktop every other day you're really missing out!

I used to habitually hop from one Linux distro to another, and I was often rebuilding my desktop every other day.

Learned a ton about distros, backups, treating machines as disposable, configuration management and scripting the desktop setup though. It wasn't completely wasted.

Just install Gentoo

Interesting, that's one distribution I have not tried. However, searching for a degaussing package comes up empty:

https://packages.gentoo.org/packages/search?q=Degauss

Until gentoo has this, I'm not sure if it's ready for the desktop. Maybe next year?

You don't need a fucking app, just echo full | sudo tee /sys/class/drm/drm1/i2c/00/power/degauss like a normal person.

Good point, I can add this to my bi-hourly disk cache flushing + cache algorithm rotation routine since I'm already on the terminal.

How do you ever find time to record a video of yourself on a white background talking about how much better you are than PC users? ;)

I have a few Linux boxes which let me regain some productivity.

I think a simple explanation could be that it is slightly harder (mentally) to move than copy a file. Windows move: ctrl + x Mac: cmd + option + v In every other program, cut/paste is the same as windows. That’s confusing - at least as a former Windows user. That said, I had many duplicate files on Windows too.

Dragging from one folder into another is also a copy operation on MacOS instead of a move like it is on Windows.

Both Macs and Windows have dragging and dropping from one folder to another on the same disk as a move, while dragging and dropping to a different disk is a copy.

Not sure why I thought it was a copy!

It defaults to a copy when dragging to external drives (holding command makes it a move)

Dragging from one folder into another is also a copy operation on MacOS instead of a move like it is on Windows.

Only if the destination is on a different volume. Like from a hard disk to a USB drive.

Hold down the Command key while dragging a file between volumes if you really want Move instead of Copy.

One of the biggest ones from past days was media files and especially photos. They are large, users like to arrange/rearrange, compress, and manually back them up (or “back them up” depending on the knowledge level of the user). They are also rarely modified after being initially tweaked.

And on the other side: Apple computers typically don’t have upgradeable hard drives so for anyone who takes photos/video it’s an inevitable countdown to zero free space.

There are many programs out there that will find duplicates files. I use this one. It gives you the option of replacing the duplicates with either hard or symbolic links.

https://github.com/qarmin/czkawka

I have been looking for something like this, thanks for sharing!

From the project:

> This name is not as bad as it seems, because I was also thinking about using words like żółć, gżegżółka or żołądź, but I gave up on these ideas because they contained Polish characters, *which would cause difficulty in searching for the project.*

Really dodged a bullet there.

Remember how it is pronounced and you can remember the spelling: Czkawka == Ch-kah-v-kah. Cz is 'ch' like Czech, and w is v. It sounds like a sneeze when you say it.

I've used fdupes and refund for exact duplicate finding.

They work shockingly well on large libraries, I've managed to check millions of files consuming several TBs in a matter of hours.

I use fdupes and it worked well, but what is 'refund' here?

ah, the auto-corrected version of 'rdfind'

In my 40+ years of computing dating back to an Apple II, I've used storage as varied as 140K floppy disks, to EMC RAID arrays attached via SANs to Sun Enterprise 4500s, to NetApp NFS clusters, to a TrueNAS box in my basement with terabytes of space.

I've been a sysadmin managing SunOS and Linux systems at scale.

I've had personal machines running a variety of OSs: FreeBSD, Linux, Windows, Mac OS, OS X, to say nothing off all the different filesystems involved across those OSes.

In all that time, I can't think of a single time I've ever bothered to de-duplicate files.

Once upon a time, you used to occasionally need to defragment some file systems. But it's been more than two decades since I've even thought about defragmenting. Every modern filesystem just handles that automatically.

What even is the use case where folks end up with enough duplicate files that it's worth worrying about?

I can think of exactly one case!

In college there was, for a glorious, brief, time, a program called mytunes. It would scan the network for iTunes instances that had the network sharing turned on. Then it allowed you to copy over entire iTunes libraries (iTunes only let you play songs from other users).

I would log on to the dorm network which normally had dozens of iTunes instances running at a go, and just copy everything to my, at the time, massive 160gb drive.

Then you would dedupe. That’s the last time I thought about deduping

I never really bothered myself, but when one of my SSDs was running low on storage, I decided to run a deduplicator over some folders.

It turns out the node_modules folders I've collected over the years had several gigabytes of duplication. The same was true to a lesser extend for Python code (because the dependency folders and venvs were a lot smaller).

I'll give you two use cases, one for non-technical and one for technical people.

Non-technical person will copy-paste a folder of documents to make a backup before editing a file. A crude form of version control. I've seen this method lead to the consumption of hundreds of gigabytes for duplicate content, because the folders contain a lot of auxiliary data like PDFs and JPEGs.

Technical person will run npm install and get a massive node_modules directory that is almost identical to the node_modules directory in another project.

For the second one I would love to have some kind of centralized node_modules used across my projects. I am pretty sure that a large part is duplicated, just for the fact that I use the same framework everywhere

One time I had to consolidate a late family members personal files spread across several types of media which ended up totalling several TBs. None of it was organized, so there were duplicates everywhere. Probably only 20% of the data was non-duplicated stuff I genuinely wanted to keep.

> Every modern filesystem just handles that automatically.

Do they? I always assumed they just try to avoid fragments but ignore them once created, and rely on bigger faster disks to hide any problems this may cause.

For folks who know they have duplicate photos, but also have a photo library large enough that it's hard to manually de-dupe them, I can strongly recommend PhotoSweeper (no affiliation). https://overmacs.com/

What's unique about PhotoSweeper is that it gives you deep control over what you mean by "dupe" and supports pro image formats like RAW and DNG. For example, I was able to automatically de-dupe byte-for-byte equivalents, then manually evaluate dupes with the same image data but slightly different metadata. Of course, it can do things like find dupes at different resolutions and keep the highest-quality version, etc. It saved me a ton of time.

I would like to know if there are similar utilities for audio files.

If your fs implements block dedupe, no need. But, it's expensive to implement and from what I see no longer recommended on ZFS.

If your fs implements hard links, no need inside one mount point. But you can't hard link cross mount.

Copies can be good or bad. It depends.

Dedup on FS works, but hard links are useful only if you (and everyone else using the PC) is aware of the hard link (and never forgets about that). Otherwise it's "I can just mess with this file, I have a copy in that other folder for backup".. until you notice it's not an actual copy.

It's better to assume all files are potentially links that must be explicitly copied before any destructive operation. Software should be designed to always produce new copies by default and overwrite files only when explicitly requested.

I'd like copy-on-write hardlinks, or deduplicated blocks, that are made automatically on a schedule as to not incur the constant overhead of dedup, but still reap some of the benefits for archival.

Reflinks are basically CoW hardlinks — supported on Linux by btrfs, XFS, and I've seen some effort to bring it to ZFS.

DragonFlyBSD's HAMMER filesystem supports on-demand/scheduled dedup using the hammer(8) command, with configurable memory and runtime limits.

https://manpages.debian.org/testing/duperemove/duperemove.8....

Sweet, thanks.

NTFS deduplication works this way on Windows server since 2012, but I don’t think any of the Linux NTFS drivers support NTFS dedupe correctly.

The behaviour is very different to hardlinks. The clone is a new file that uses the same blocks. When you modify the clone, writes are written to new blocks rather than the old ones.

So if I have a disk image that takes 1,000 blocks, and clone it - I now have two images that total 1,000 blocks. And if I modify one block in one image, I now have two different images that total 1,001 blocks. With a hardlink the modification would affect both images.

The way I visualise the difference, is that a directory entry points to a file entry, and a file entry points to a list of blocks. A hardlink is a new directory entry that points to an existing file entry. a linked clone is a new file entry that points to the same list of blocks.

> But, it's expensive to implement and from what I see no longer recommended on ZFS.

Btrfs (and more recently also XFS) supports on-demand, batch dedup which doesn't have any of the downsides of ZFS's always-active dedup. This is an implementation issue, not a categorical problem of block (or extent) dedup.

Btrfs is like an almost perfect file system. If only it had proper stable support for software RAID. It's weird that it still doesn't have it, I thought enterprises cared about that.

It has proper stable support for many kinds of software RAID: https://btrfs.readthedocs.io/en/latest/Status.html

Except RAID6 which is the one I want. From your link:

https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5...

Also, reading stuff like "mostly OK" in a file system doesn't inspire confidence.

The ideal situation here would probably be to update any identical but not reflinked files to be reflinks. But if there is no API like you suggested then it may not be easy. Also ideally there would be an API to do this atomically please share the storage of these files assuming they are the same)

If Time Machine is tracking file contents for deduplication it would be really cool if it did this to the source files whenever it noticed that they were identical.

So if you change one byte in a clone, then it will just update a single block in the changed file while leaving all the other clone's data streams unchanged. But what happens if you add (or delete) a byte near the front of a 10GB file? Does that cause the entire data stream to be written, since the rest of the data must be shifted up or down?

Yes, but precisely because of that there is a bit of resistance to people using that operation on large files, so it is not a common thing. It wasn't that long ago that rewriting a 10GB file from top to bottom could easily take over three minutes (50MB/s write speeds to a spinning rust hard drive = 200 seconds, and if we account for also reading the file it will be even slower), so people tend to learn to program without asking for that operation.

While there are many people who weirdly (to me at least) grotesquely overestimate the value of deduplication, it is legitimately useful because we tend not to want to completely rewrite large files very often like that, so even modifications to that file are very likely to either be appends or modifications of something in the middle without shifting the entire file.

I used to dedupe a lot to save space, on any OS, Mac, Windows, and Linux. But one of the main reasons I end up with many dupes is various archiving and backup schemes I use, and I’ve realized deduplicating reduces backup safety. If the source file gets somehow corrupted for any reason, all copies are lost. I’m sure a deduping file system is slightly better here than a deduping utility and/or my own scripts, but still having only one copy of the bits is more precarious than saving multiple copies.

Back in the day I used to do this on file servers at the block level. Was valuable as storage was more limited, IT would often have duplicated large deployment images and users would often have duplicate copies of the same files. Multiple one user by ~1000 and you have significant space savings.

I use btrfs + bees (https://github.com/Zygo/bees) for such cases

>Finally, why don’t some housekeeping utilities recognise clones? That’s because there’s no official way to tell them apart.

Is it intentional?

If there were any official way to tell them apart, would they still be clones?

I would prefer not to.

Reply on news.ycombinator.com