Hacker News Reader

Or let users help you identify such pages. Instead of showing a vague message that indicates temporary failure due to time out, show the 404 with option to report the broken link.

I miss picks because that's how I kept track performance of results from my own algos and compare with supposedly experts and crowds.

Why write a bot? Doesn't yapache log such events? Just run a Hadoop job on the error logs.

I'd run it across the entire site. Sure, we do have logs, but it's not easy to access them for all *.yahoo.com. Logs from each property are ingested into different places. The infrastructure is way more complex than it should be making bare-minimum monitoring like alerts on non-200 responses on all properties difficult to maintain.

Shouldn't the yahoo.com servers already log when they return error responses? :)

Each property has their logs ingested, indexed, and archived differently. I only have access to my little arm of the company infrastructure. Logs should be monitored properly by the responsible owners of each property (or in a more sensible way), but unfortunately this is not the case... :)