When you have financial incentive to build your business on someone’s data and you scrap literally millions if not billions of pages - it’s unethical.
This data is often of great public value. I track conversations around a social issue as part of my work for a non-profit.
I'd counter it's unethical to prevent people from accessing this data.
> great public value
Having been to twitter mostly through the most recent prominent war, man the signal to noise ratio is really low even when being careful about who to follow and who to block. There is so much disinformation, bad takes, uninformed opinions presented as facts, pure evil, etc.
So I guess it could be used for training very specific things or cataloging the underbelly of humanity but for general human knowledge it’s a frigging cesspool.
I do not use the Twitters myself, and actively discourage others from doing so. Sends people bonkers.
We were actually gearing up to switch to paid accounts as we found use cases that could subsidize these efforts... And then the starting price for reasonably small volumes shot up to like $500k/yr.
We have robots.txt. If Google doesn't respect that, it's unethical. Don't you think so?
Not that the people you want to respect that would
Especially since they're not moderating things or anything.
Agreed. However, it's probably covered by their terms of service.
Same thing with the recent reddit kerfuffle. I'd have much preferred a Usenet 2.0 instead of centralizing global communications in the hands of a handful of private companies with associated user-hostile incentive structures.