The information is available to the public, just not for certain classes. This is and should be legally unenforceable.
If something is truly meant to be private it should not be referenced from a public-facing page or it should have access control enabled.
"If you truly didn't want trespassers you should've put up a gate."
As a property owner, a no-trespassing sign won't protect you from the lawsuits that result when a toddler drowns in your pool. You're expected to do more (like putting up that gate).
Equifax's systems are peppered with "no-trespassing" motds at login. They also have a robots.txt file. We expected them to do more.
Same for leaving keys in your ignition, guns unlocked on your nightstand, etc. "Don't touch" signs won't absolve you of responsibility when either gets stolen and used in a spree killing.
So yes, as the owner of any sort of asset, in most contexts it is your responsibility to implement access controls to keep unauthorized traffic out.
Good analogy. I wonder i operating fa poorly secured website that leaks private information could be seen as an 'Attractive Nuisance' [0] and the owners could be prosecuted for that, rather than the hackers!
0. https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine
Even for those who think that robots.txt should be enforceable, allowing some bots but not others makes it difficult for a new player to have the same equitable access to information as the big players.