>now we were having to look through multiple copies of the same file only to determine they were all duplicates and so forth.

That sounds like a old fashioned ways in front of a problem, and in front of a solution.

There are multiple automated(or even just "hand" sorting) solutions that could chew through the duplicates.