FWIW if people are interested, I wrote https://github.com/karteum/kindfs for the purpose of indexing the hard drive, with the following goals

* being able to detect not only duplicate files but also duplicate dirs (without returning all their sub-contents as duplicates)

* being able to query multiple times without having to re-scan, and to do other types of queries (i.e. I am computing a hash on all files, not only of those with duplicate sizes. This makes scanning slower but enables other use-cases. N.b. beware that I only hash fixed portions of files for files>3MB, which is enough for my use-case considering that I always triple-check the results and is a reasonable tradeoff for performance, but it might not be OK for everyone !)

* being able to tell whether all files in dir1/ are included in dir2/ (regardless of file/dir structure)

* being able to mount the sqlite index as a FUSE FS (which is convenient for e.g. diff -r or qdirstat...)

Still work-in-progress, yet it works for several of my use-cases