Hacker News Reader

You may find the amp pages slightly easier to scrape.

i.e. This: https://www.bbc.co.uk/news/amp/health-54795657 vs This: https://www.bbc.co.uk/news/health-54795657

I'm working on a similar thing at moment (BBC html -> markdown) so also exploring the best way to do it.

nice thanks, that looked promising but I've checked a number of older articles and they have no amp version (despite the HTML for the site referencing an amp version, the url is 404!)

I'll see if I can figure out if all newer pages will have permanent amp versions, or whether all amp versions drop away over time.

Reply on news.ycombinator.com