it can detect duplicates in your collection because it maintains a small database of what it's seen. Great for ingesting the big pile and cleaning up