If someone I didn't grant access broke (in a very smart way) into my house, turned on the lights for a minute and then left, I'd still be pissed and would call it illegal.
(Somewhat tangential, the "networks as a 3D space you travel around in with locations you visit" analogy does more harm than good. It's not what's happening and it results in muddled thinking.)
[1] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act