For duplicate files I've used the following: - https://github.com/adrianlopezroche/fdupes - https://github.com/pkolaczk/fclones

The latter works well for larger datasets, outputs a TXT which you can analyse and decide what to do with.