So HN itself does the merging and the raw dataset still includes the numerous duplicate submissions then? If this is the case it's not just sources with a lot of content like medium.com, github.com, nytimes.com being dragged down, it's any popular source.