Hacker News Reader

This is easier than deduplicating the many different URLs which have the same content. A harder problem awaits you!

ML & basic stats

What approach would you be going for initially for deduplicating same urls?

You might look at a real data processing system, like something from the Apache projects