Ha! If you know a way around them I'd appreciate the tip.
(1) Try this URL instead: http://www.nytimes.com/pages/
(2) Hit the URL by date: http://www.nytimes.com/indexes/yyyy/mm/dd/
Example: http://www.nytimes.com/indexes/2011/12/03/
You can also use this URL structure to get the Homepage back several years (2001) as it was around midnight of that date.
http://www.nytimes.com/indexes/2001/01/01/
http://www.nytimes.com/indexes/2001/09/11/ (Notable)
Not sure if this (or any other section) is of interest too:
http://www.nytimes.com/indexes/2010/12/03/todayspaper/
(3) When you scrape the page find out the link it provides to the Homepage and then try that. I had some success doing that.
What I really want is THIS:
"Reward - NYTimes Login Script" http://donohoe.tumblr.com/post/10723388191/reward-nytimes-lo...
which would get around that problem.
NYT homepage editor has flexibility on when they "roll" the date depending on news events (and content can be ranked ahead of time too). But yeah, basically midnight.
Internally I know editorial preserves them on an hourly basis. You might ask current NYT-ers if you could get access to that?
I know I did a scrape every 10 (or 2?) minutes of the Homepage HTML for a couple of years (2007 to 2011?). If I can find that data and its still meaningful I'll get it to you. It was quite a few GB.
http://www.nytimes.com/indexes/2012/01/03/
... a misstep we later fixed. For what it's worth, there's also an internal version of the homepage archive that doesn't suffer from this problem, and is snapshotted hourly.
Can your approach accept cookies?