We essentially do do this--duplicates are only looked for in the (project, user, time) shard. It’s certainly more than a handful of pages, but nothing extremely cost prohibitive at all. The indexer dedupes a whole shard at a time, but only if at least one duplicate exists--in the normal case, we just do nothing. Unfortunately, we also can’t control the number of duplicates that we get (since we ingest data from our customers, we can't control how good it is), and a lot of the time, the number of duplicates are fairly frequent due to suboptimal implementations of Mixpanel, making the whole shard deduping a fair tradeoff. We definitely never touch the network :slightly_smiling_face: