And if you're going to bring up the UserAgent spoofing, let me remind you that most browsers have done something like that for > 15 years.
> If the server owner had so desired, they could have made the data private by adding a password.
But the server is still just sending data in response to a request, even with a password. The only reason a password is a line we draw is intent. It's hard to say you didn't realise that guessing at someones password was wrong.
A logical step is to make that machine readable. Oh, wait, suddenly this is getting to the server software and configuration, that server developer/administrator had screwed up.
My question is - why we don't make that logical step and simplify things instead of relying on some "should be common sense" and "you should've known you wasn't supposed to do so" completely-gray-area?
You mean like the Terms of Use for the AT&T website?
As for those exact terms, I suspect (IANAL) those exact terms prohibit almost any access to the site, as, for example, they forbid any programmatic access to obtain the information, and I haven't heard of any non-software user-agent implementations.
As opposed to a human being in a non-programmatic way, opening his browser and accessing the website.
What's so hard about it?
Doesn't, for example, Firefox, perfectly fit this description? Yes, I do manually enter the base URL to access, but if that's the distinctive feature...
> As opposed to a human being in a non-programmatic way, opening his browser and accessing the website.
... then manually typing in ./scrape.py www.att.com is non-programmatic, too. :)
Or, maybe, I'm not getting the correct meaning of "automated" due to bad English comprehension and false analogies from other languages. But I always thought every request on the Internet is automated and done by some kind of hardware+software combo, so forbidding "programmatic" access is complete nonsense (access control and rate-limiting are the proper solutions).
(And, if that matters, author of scrape.py does not need to conform to AT&T's TOS if s/he don't actually use the script by themself.)
What if I set up a website, put a clause saying "you agree to pay $50/page view" in there, and hid it away. Google crawlers will find my site in no time, and then I can start raking the dollars in, right?
He did exploit the fact that AT&T did not make the endpoint in question accessible only if the logged-in user matched the actual user ID (or just made it entirely inaccessible).