> "So you want to scrape like the unethical boys?"
What's considered ethical is a very debated topic.
An assertion that something is simply "unethical" should be seen as the starting point of a discussion, not as a self-evident fact.
What's considered ethical is a very debated topic.
An assertion that something is simply "unethical" should be seen as the starting point of a discussion, not as a self-evident fact.
I find it really hard to see how you could twist ignoring this clear lack of consent, and going to great lengths to circumvent what was clearly put into place to prevent you from doing the very thing you are doing, how you could twist that into an ethical action.
It may or may not be technically illegal to do, you're but that is not a statement about what is ethical.
I run a search engine and an internet crawler. I do this all the time. To this date I've never had a webmaster that didn't permit my crawler access when I've asked nicely.
OK less provocative, you have new algorithm to identify inaccessible websites, your automation is scary good, crawling a site you can identify many issues that most sites would have to pay for a full audit to get, but now these sites have problems - if you can identify their sites as being inaccessible then they have to fix these problems due to various accessibility standards that apply in the regions they operate in. But if they don't allow you access then they can maybe make an argument they are accessible due to audit they did last year, at any rate they don't want to be forced to spend money on accessibility issues right now which it sounds like they might have to if they let you crawl their site.
Version 2 of above, some years ago I spoke about a job with a big time magazine publisher in Denmark and said one of the things that would make me a good employee is my knowledge of accessibility and their chief of development said they didn't have anyone with disabilities that used their site - so if I ask that guy to crawl their site why say yes? They have no users that would benefit!! Stop abusing our bandwidth bleeding heart guy.
Bullying websites into accessibility compliance will most likely lead to them following the letter of the standard without giving a second of thought as to whether the content is in fact actually accessible. It's very difficult to get someone on board with your cause if your initial contact is an antagonistic one.
I scrape and process websites of actors engaged in fraud. I do this to make the data more presentable to the proper authorities and to help uncover further evidence of their activities.
I suspect that asking for consent would be quickly denied and the data/evidence would quickly become inaccessible.
Is Marginalia opt in, then? Surely "not having a robots.txt" ("you didn't say no!") does not equal consent. And surely you could just ask all the webmasters you are scraping from for permission, since you have noble intent.
My point is that this is just hypocritical; you are placing the moral boundary right below what you are doing, while claiming moral superiority. If you ask others (e.g. anti-search Fediverse), they would think you are immoral too.
Both of these are the same?
The property websites are incompetent to solve the problem, or don’t care, but either way they sure don’t want you scraping their valuable data.
Is it still unethical?
Try asking a startup for free software licenses or seats or whatever as a non-profit. "We're entitled to 40 seats of your SAAS solution because we're a non-profit working to solve world peace." It's definitely within the startup's pervue to respond with a no.