Scenario one: you use software to connect to their server and download a webpage. You are a user.
Scenario two: you use software to connect to their server and download a webpage. You are a "bot".
Make it make sense
Scenario two: you use software to connect to their server and download a webpage. You are a "bot".
Make it make sense
Even by the typical robots.txt definition, a bot has to operate more or less fully autonomously. If the user initiated the request, it doesn't count, and it shouldn't follow robots.txt.
Are you implying that's a bad thing? Doing things for social status has been an integral part of society for millennia. It's a legitimate motivation that can benefit both the recipient of the status and the rest of society. In this specific case, if you assume that the author makes content that is useful to you only because of the social status reward, taking away that reward means the author will stop making content that is useful to you.
This is a change. People put up new caching layers, and it urks people running a website in a single small machine. Yes, this could always happen with slashdotting but it's different now.