- I definitely see bad design in interpreting the RFC as "if no caching metadata, then cache forever;" this violates Principle of Least Astonishment.
- Also, "301 and 302 has completely different meaning to a crawler" seems illogical.
- Perhaps the RFC should have also specified something like "in absence of caching headers, don't assume 'cache forever', that is a long time"; it does not try too much to prevent implementors from shooting themselves (and users) in the feet.
In defense of the browsers - the RFC does say, in essence, "rewrite the 301'd URL and be done with it," so the browser does not even need to deal with the redirect any more.
At the least, a redirect should not be permanent across name server changes (when the domain changes hands). Unfortunately, this would effectively be adding state to HTTP.