Hacker News Reader

Comment by softwarelimits | Hacker News Reader

Does anybody know of a good deduplication script / tool that runs on every filesystem and can be used as a nightly cronjob?

View on HN

Here's a quick one if your filesystem supports hard links:

    #!/bin/ksh

    typeset -A hashes

    find . -type f  -exec shasum -a 512 {} \; |
        while read hash filename; do
            if [[ -z ${hashes[$hash]} ]]; then
                hashes[$hash]="$filename"
            else
                echo ln -f "'${hashes[$hash]}'"  "'$filename'"
            fi
        done

Pipe to /bin/sh if you're happy with the results.

Good answer, but unless you mark the files read-only/immutable, this ought to have big red flags all over it. Userland just can't do this safely in the general case.

Good point. This is for data that's not in use.

Do you actually use this? Seems at the minimum you want to use -xdev with find and do something better than ln -f.

Yes, I forgot to put in -xdev, but what's wrong with ln -f?

fdupes -rL /dir