Creators of the data quality tool for computer vision, fastdup, continue to improve on their free release https://github.com/visual-layer/fastdup

Here's a short video of some recent results for LAION 400M https://www.youtube.com/watch?v=dlRCm29Upu4