And if I was naked at the time, and if they then sold those pictures of me, I'm fairly certain it would be a crime.
If you setup a directory /admin on your webserver, and forget to password protect it, and someone accesses it, you can't really call it hacking.
Whether or not accessing anything on a webserver is illegal depends partly on your intentions while doing so. That's a basic function of most court trials--to discern your intentions. For example, see the differences between the various degrees of murder, and the way such trials often hinge on "preconceived intent."
If my intention in accessing "/admin" was to access your server without your authorization to access some protected piece of information, damage computers, or defraud you, that may indeed be a crime. See http://www.law.cornell.edu/uscode/18/1030.html . It doesn't matter whether it was password protected or not. If I went there by accident, however, it probably was not a crime, although depending on what I accessed I may wind up having to go to court to prove that my intentions were benign.
My point which you gloss over is in the definition of "attack".
Whether browsing to a publicly accessible url with no security is an attack or not, is a very grey area.