A password works by there is a piece of text that only you know, and thus, when you give that password to a server, that server knows who you are.

can't you also view a url as a password? (If only I know the url, then only I can download the file).

I am able to give out a url to someone else, so they can access the file, likewise, I can give out my file server's username and password, and whoever has it can also access my files.

I think it is primarily a legal concern. Legally, caching public content is not always an acceptable (or at least clear-cut) practice. I concur in thinking that's a little silly, but enough people disagree for it to be an issue that Google has to take into account.

There have been similar complaints about attempts to aggregate the contents of Hacker News posts, for example.