All websites are private (excepting maybe government sites). In most places the internet infrastructure itself is private.

You're conflating a legal concept that applies to areas that are shared, government owned, paid for by taxes, and the government feels like people should be able to access them.

The web is closer to a shopping mall. You're on one persons property to access other people's stuff who pay to be there. They set their own rules. If you don't follow those rules you get kicked out, charged with trespassing, and possibly banned from the mall entire.

AI bots have been asked to leave. But, since they own the mall too, the store owners are more than a little screwed.