Hm, I disagree. Either information is public, no matter for who. Or the information is private, and you should have ACL for accessing the information. I don't think it's fair to say that information is public if you're a human but private if you're a machine, or vice versa.
It's not about if it's difficult to build but rather the principle behind if you can just allow humans to read something.
And that bots don't lead to revenue depends on why the bot is navigating on your page no? If it's some indexer that links back to your website and it's a popular index, then you'll maybe end up with more revenue thanks to that bot than a normal user.
Your arguments about revenue are website-dependant and it's the website owner who is in the best position to decide whether robots are good for them or not (and plenty of sites don't ban bots in their robots.txt). In this case, the company that ran the bots is directly competing with Linkedin's products that sell aggregated data to employers and such, and linkedin clearly decided it's not going to lead to more revenue for them.
Does an ad-blocking browser counts as a bot or as a human? And what is something that concatenates all of your infinite scrolling to represent a paginated view? What is something that changes the structure of your page? What is something that concatenates different pages before displaying?
I would definitely want some intent provisions in, but saying something is accessible therefore free game seems too wide.
The problem with analogies is that many equally valid analogies that can be made, but with many different points. I would argue that the real life equivalent is "Have this free book, but you may not read Chapter 4."
If I put on my website terms of service "please don't try to go everywhere" , and then you do... seems like you did _something_.
I don't really get what sort of stuff is enforceable, though.
the ACL is the robots.txt. A door with or without a lock doesn't determine whether the place is public or not.
One can't post a sign in public that tells people not to look at other publicly visible signs and expect the government to arrest or fine them for ignoring it.
What if I user curl to pipe web content to my mail so that I can read it in a quirky way? What if I write a Chrome extension to crawl a site? Where does w3m stands?
This is not a question of the tool (UA) but of the intent (mass crawling, indexing, mass-replicating stuff). robots.txt is made as hints for crawlers and the like, not optimistically ACL whether something is public or not.