"Looking at our deduplication function, it’s functional and performant, but not very descriptive. Essentially it checks a running set called seen, which is checked for originality as the deduplication script progresses. As python Dicts are essentially a hash map, the in check compares element hashes instead of each individual component of the dictionary object. This massively speeding up the deduplication process."
Uhm..I don't think so.
It is just extra work to check if the objects are already in the set vs just stuffing them all in and let the set handle uniqueness.
Am I missing something?
https://gist.github.com/pstoll/ae73582763540051d321a4eb15304...
Again, a minor thing. But seeing something like that makes my 'what else do I need to review' detectors go up.