RobSm

You can always stop bots. Add login/password. But people want their content to be accessible to as large audience as possible, but at the same time they don't want that data to be accessible to the same audience via other channels. logic. Bots are not consuming your data - humans are. At the end of the day humans will eventually read it and take actions. For example chatgpt will mention your site, the user will visit it.

And no, nothing was different before 2022. Just look at google, the largest bot scraping network in the world. Since 1996.

The reason is different. When you need to scan 5 million IPs on 22 port is one thing, but 5 million IPs times 100 or 1000, then you simply run out of resources and since 99% of people use 22, why bother?

There are at least 1 of us!

Looks weird :)

You need to upgrade your knowledge. Use only [2-9]. That's it.

Google scrapes everyones data yet you use Google every day and find it useful.

No

People don't care about such content and so does Google.

Explain more about it

And if I open the website in my browser and then copy from browser to my computer, then all is good?

So in the scale of google, 'not many' would be some few million per month? And all is good then, right? Even you use their scrapped data probably daily and are totally fine with that, right?

You think google bots read contracts before scraping website? really? :) If you had any experience in creating websites and launching them online, you would know how fast and often they arrive and how they do not care about your TOS. So the real 'violation' numbers might be very scary...for you.

https://ironcladapp.com/journal/contract-management/are-brow...

And if you manually copy someone's data they worked hard to generate to go and resell, then it's ethical?

This is so exactly. People do not realize that when they use chrome to view website, chrome is their 'scraper'.

And the goal of webs craping is not to get illegal data, but to have efficiency and performance by not doing something manually but letting computer do the repetitive tasks. It's a productivity tool. You can't make something illegal just because it's an automation instead of 'manual' operation.

No. robots.txt is not something that is defined and enforced by the law. Just because someone came up with some 'recommendation' like robots.txt does not mean this is the law

If you post that data on a public domain, that is publicly available. It's like writing that info on a cardboard and putting it in the town square and then saying 'why you people steal my data!'

How many contracts google breaches scraping billions of pages every month?

How one goes about using TLS proxy? Are there any services? You have any links? Thanks.

What's the point of TLS proxy?

"and offered it to everyone for free, making a fortune" - this makes no sense

Thanks, go it. You can delete now

""

I get it why someone else scrapes it. But why customers upload data in the first place? Aren't they interested in getting some OTHER data from you and that OTHER data may as well be scraped?

Makes little sense - customers upload data to you and they don't want any data back? Really?

Building API is 5 times easier than building routes for your public webpages, which is basically an 'API' as well.

If you build your site in a way that multiplies each request 10x, well then that's what you get. Don't do that and you won't have issue with requests. Or handle those requests properly. There are solutions to that. You know how many requests your local google CDN gets? They know how to manage load.

Exactly. I am surprised that the 'devs' can't figure out a way to block only annoying/excessive scrapers. Most likely they are just lazy and then just put 3rd party 'solution' and job done. Pay me.

And how do you get your 'inventory data'? Aren't you scraping (or using scraped data) yourself? Oh the irony :)

Why do you think those bots were scraping your data in the first place?