Equating entering buildings with communicating with computers on the internet really is an awful analogy. You can stand in front of a building and be able to tell whether it's a house or a store, i.e. a building meant for private access or for public access. You can also tell a difference between a back door and a front door by looking from afar. You can do neither of these things with computers. You can't look at it from afar to give you clues, you need to communicate with it. The way computers communicate is dictated by protocols. Protocols will tell you stuff like whether you're allowed to talk to them or not. TCP includes telling you whether you're allowed or not via its protocol. HTTP will tell you whether you're allowed or not via its protocol. If the protocols don't tell you they're unwilling to talk, and continue by talking to you, you can only assume its ok for them to talk to you.
When you first try to communicate with a computer, you can't even know it exists until it replies to you. For the analogy with entering buildings to hold, everybody must be blind and deaf and all buildings must be the same from the outside. Under these conditions, you need to lock your doors, because the only way for anyone to be able to differentiate a house from a store is whether or not the door is locked (TCP connection accepted or rejected). When they approach a door, they can't even tell if the door is really there. They might just grasp the air when they reach out with their hand (TCP timeout from lack of response).
A better analogy is people talking. Everybody is still blind but not deaf. Let's say your robot slaves are talking. Your robot, probably bored, calls out to somebody, "Robot 10?". A robot replies, "yeah?". So, now you know they exist and they're willing to talk to you; you've initiated a TCP connection. "So, how's it going?" your robot asks; HTTP GET /. "My master got married last week.", he responds; HTTP 200 OK. Then comes out his master from behind the curtain, and says "No! It was never my intention for my robot to give out this information. In fact, it was never my intention for my robot to reply to anything anyone ever said. This is your fault!", pointing at you. "You called out to Robot 10, and he replied when it was never my intention for him to reply. He should have said, 'Sorry, I don't talk to strangers' (TCP connection rejection or HTTP 403 Forbidden) or refused to talk (TCP timeout from lack of response) or something. I could have told him to keep quiet, that such things are confidential, but... but... but you should not have called out to Robot 10! You're a criminal! Don't ever do that again. I may just have configured him incorrectly to die whenever he hears a greeting and that will be your fault too if you greet him! I'll charge you with murder for greeting him! and I'll sue you for compensation for the damages I incurred from my robot not being able to do some work for me while being dead."
We could disregard a computer's configuration as indication of their master's intent. However, that doesn't mean not entering someone's house via the back door. It means not talking to anyone ever for fear of them turning around and accusing you for talking to them or for hearing stuff they willingly told you.