I run something I call an "ethical crawler". It’s designed to avoid being a burden to websites - it makes requests very infrequently. Crawling the internet reliably has become increasingly difficult, as more and more content is protected or blocked. It’s especially frustrating when RSS feeds are inaccessible to bots.
404 definitely are not a problem for me. My crawler tests different mechanisms and browser headers while exploring the web.
My scraping mechanism:
https://github.com/rumca-js/crawler-buddy
Web crawler / RSS reader