It sounds great but it is a completely ignorant thing to say.
- trying to disrupt business processes (eg: false referral listings, gift card scams, etc)
- trying to disrupt systems
I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs that become overhead. Bots are usually coded with retry mechanisms and ways to change connection criteria on subsequent retries.
- Reselling aggregated data
- Competitive pricing and inventory data
- "Sniping", like with auctions, event tickets, or things like airline check-in processes that are first-come, first-serve
- Weird SEO stuff where people scrape content in the hopes that isn't indexed yet, and they can beat you to it.
- And, sort of in the space you mentioned, searching for existing vulnerabilities by various signatures, or trying to brute-force guess things like passwords.
I know this is off-topic, but I'm really curious. How does scraping the web help with home automation? Maybe downloading weather data could help, but crawling the web? I think I'm missing something about home automation.
I'm sure there are others, just from the top of my head:
* Electricity prices (as OP mentioned). Especially for people with solar panels or multiple options for heating.
* Watching for availability/prices of products or new homes one might be on the lookout for. Notifications at price drops/availability
* Public transport: next bus/trains from closest station, delays and interruptions
* IMDB/tvdb/etc for monitored shows and movies. Common with sonarr.
* Air quality, covid outbreaks, whatnot
Imagine if the goal is images and videos, now you've got yourself some heavy duty scraper that could cost the website owner lots of data fees.
So you do some thing which once a day scrapes a site and pulls off some data that you use in your thing. Maybe you talk about it to friends, or you have this thing as one of your github repositories. Some of your friends download the repository and also start using your thing. They talk to people about how cool your thing is, or what it does and the nice convenience of automating something that you used to have to do manually.
There are 86,400 seconds in a 24 hour period, probably folks won't change your code at all at first, and as it diffuses into the community some webmaster starts seeing this weird spike of queries that happen once a day at some time. Different addresses but always the same kind of request.
It's not a problem when its like 10 or 20 qps burst but when it starts getting up to 100 - 200 or worse 1000 - 2000, it causes the system to perhaps spin up additional instances that it isn't going to need after the burst and waste money. So the webmaster starts denying those requests with a 404.
Now sometimes your code works and sometimes it doesn't but you don't know that the webmaster is fighting you yet. Maybe eventually you start randomly varying the request time, or the people who have copied your thing are in more varied time zones so you it starts getting spread over the day.
now the webmaster is seeing bursts of traffic nearly every hour on the hour and that is weird so a more aggressive mitigation strategy is enacted.
People using your thing complain that it keeps breaking so you look into it and realize that the site is trying to block your requests. Perhaps you don't understand why this is, or perhaps you do and don't care, either way you come up with some strategies that avoid the block (maybe your rotate the user-agent or something).
Now the query traffic is spiking again and the webmaster is getting complaints that this 'bot traffic' is resulting in useless AWS fees because it isn't part of the revenue traffic and it is forcing the service to add more resources for their customers.
Not all scrapers are malicious, but my experience is that it is rare that a non-malicious scraper application isn't talked about and shared (amongst people who have a similar itch that the thing is scratching) and because its all open source it spreads around.
That's exactly my case though. I have a few scraper scripts that I've never published. So what if it's rare? Do I deserve to be treated like a botnet just because it's inconvenient for some webmaster or company to do otherwise? That's not fair at all.