Hacker News Reader

Comment by mmcclure | Hacker News Reader

I can't help but feel a little conflicted about things like this in the current climate. On one hand, as a developer trying to just get some updated stuff into a spreadsheet, this looks extremely useful.

On the other hand, there are a lot of people right now that want to keep stuff accessible to humans and not have it scraped for models. I know the lid's completely off the box on that front so it's probably useless to fight it, but seeing products explicitly designed to circumvent bot prevention feels kinda bad.

View on HN

Every single time I see these scraping discussions I get the same thoughts:

Businesses use data from the user. The Business does additional crunching on that data to derive new interesting data for the user. Who owns the data? The user or the app?

At the very least the user partially owns the data and as such, I'd argue that the user should have the right to share the data between different applications however they see fit. However, businesses tend to think that they somehow have the legal (moral even?) right to keep that data in their walled gardens. For as long as this (imo unfair) stance is common, I think that data extraction by use of these anti-bot-bypassing technologies is fair game.

the growing IP centipede is an interesting GIGO dilemma

If the data is public or semi-public then chances are I consented to display that data there. I consented to it being used on that site, for the purpose of that site. Not for random other companies.

And most scraping isn't done by users. It's done by companies. For profit. Often for less than enlightened reasons.

LinkedIn is a good example: I want my data displayed to people on that job site. I don't want it harvested by every recruiter under the sun who will then spam me. I certainly don't want that data sold between those recruiters long after I deleted my account on LinkedIn. Tinder and sites like that are also an obvious example: yes it's (semi-)public, but I also wouldn't want it to be scraped and harvested by some company – I just want it to be shown temporarily to a limited set of people.

In general, I don't think people should have a moral right to decide where and how the data that they made public is used, or to decide if it can get scraped or not.

And, in general, I take the fact that you published something on the Internet as a tacit moral consentment for the rest of the world to use it how they want.

This comes with a couple of big asterisks, because (1) Copyright law exists, and I generally try to not break the law, even if I don't agree with it. But the discussion in this thread is mostly separate from copyright: for instance, I don't think a court would see someone scraping and redistributing data from someone's LinkedIn profile as a copyright infringement case.

And (2) because I think that in some specific cases, using published data can be morally wrong, but not as a general rule.

i somewhat agree; people volunteer it when posting anything online. but they also volunteer their advertising id on their phones (even if they dont know it) - just as they dont know (and dont care) they are the product when on websites like facebook

i feel the 'antibot' stuff is more related to the adtech industry vs site-scrapers - remember getting a dedicated server and having friends click on links just to pay for it? Geocities and all these free websites, the biggest costs were bandwidth and storage (not that its not now)

since the AI Boom, there's just more hype over people wanting 'credit' (or money) for something they posted on a forum X-units of time ago.

its called the World Wide Web for a reason.. keep it open, even if it is to 'a bot' - never know when somebody's 'bot software' is reading your webpage for somebody who has some disadvantage and needs assistance

Agreed, we've seen a lot of governing bodies _now_ start scraping to fight scam sites and "fake" sites, so there's a lot of legitimate use-cases popping up. It's not necessarily just about gathering e-commerce data or backfilling lack of an API anymore.

Source: Founder@browserless.io

So how will you ensure your service is only used for "good"?

no services remain "good" so why ask the question. it gets ignored or answered but then a decade or two down the line the philosophy gets scrapped, a story as old as time.

I still don't understand why Google publicly removed "Do No Evil" from their company. It's one thing to not follow it but to actively remove it feels weird. Like they wanted to say "HEY Y'ALL GET READY FOR EVIL!"

You can go to https://abc.xyz/investor/google-code-of-conduct/ and ctrl+f "evil", and you'll see it's still there.

Can confirm this Google document contains the word "evil."

Interesting I remember a bunch of articles and stuff talking about them removing the do no evil bit. Guess I didn't do enough research. Appreciate the correction!

Same way the cooking-knives company ensures those are only used for cooking.

I fail to see any legitimate use case for getting around protections that web dm developers put in to help prevent this.

I know this is slightly off topic, but I have to ask: Why not create a new account rather than use the same self-described throwaway account you used to ask about raising prices (for browserless?)[1]? For what it's worth, I initially assumed you likely weren't legitimate and didn't engage, so if you want to keep contributing to threads as yourself you might want to consider using a new/different username.

[1] https://news.ycombinator.com/item?id=21271224

the problem isn't scraping, the problem is the "content" that is being "created".

A site I worked on changed our captcha, and in the 24 hours after, we had a 90% reduction in new accounts. We had a full API including webhooks, the only private data was on the account page, there was no reason to scrape, but even if you did, there was no reason to be logged in... these accounts were used to harass, intimidate and scam legitimate users.

On the other hand, I'm not sure if the whole idea of people visiting a website where they ought to behave and play by the house-rules is doing the world much good. And I'm not just saying this because one of my pet projects was crippled by an overzealous bot-prevention system (yes I was using a program I wrote myself to access your system, no that doesn't make me a bot).

Quite a few things would be vastly better when the information that is on websites is not just publicly readable but publicised in the full sense of the word. Several industries are paying 20~40% revenue just to be findable.

Now maybe scrapers aren't the best solution, but making stuff less accessible is usually not doing much good. Especially not for accessibility.

> keep stuff accessible to humans and not have it scraped for models

I suspect that over the next 100 years, as AI and biotech come together, the boundaries between human and bot will start to blur. At some point it will be unethical to differentiate between human, bot, and everything in-between.

I argue that that time is now, if at the very least for the sake of AI-based assistive technologies to be developed to turn a disabled {100% human} into a highly functional {80% human + 20% bot}.

Bot detection should, for this reason, be doomed as an ADA violation.

I agree, being a dev myself hosting few website, and a data enthusiast, it is very hard to balance both, except if all data was open source and released for all it's owners. All online platforms, especially social media relay on users input (reddit, HN, Twitter, etc ...) yet the trend seems to be heading the other way (closing down apis, etc ...), imo this is quite unfair. Not to mention awful practices by shopping giants like Amazon and Aliexpress, Airlines, Booking.com (et al.) to profile people and "scam them". The data ethic question is lost on both sides, only the few decent ones that remain are the true victims, you mom'n'pop website.

It's community hostile. The problem is we don't consider the internet a community.

That's incorrect. The fact that the internet is not a community is not a problem - it's objectively correct and morally desirable. It's neither possible, nor desirable, nor good, to treat the internet as a community. It isn't, and we wouldn't want it to be in the first place, just like we wouldn't want all the nations of the world to be fused together.

It may not be now, but some people remember the time when the internet felt more like a community and it was glorious.

That was only possible because the internet was far smaller. There are now billions of people with internet access - the cat is out of the bag.

The best you can do is try to form your own little community in a corner of the internet - which is actually a great idea. Mass social media has so many problems - trying to make a little nook for yourself and a small group of enthusiasts/friends seems like the best way to go.

Perhaps the "conflict" lies in the notion that a "developer trying to just get some stuff updated into a spreadsheet" is somehow different from "people right now that want to keep stuff acceessible to humans and not have it scraped for models [for use by developers]."

What if all users of the www were treated equally. There is in practice a conflict of interest between www users calling themselves "developers" and the rest of www users. Yet these "developers" purport to act 100% on behalf of the rest of www users, not themselves. The failure to acknowledge any self-interest is dishonest.

The problem of scraping content for models should be solved on a legal level. Trying to do it on a technical level is mostly futile, and wrong, because it actively makes legitimate uses worse.

If you really really really want a technical mitigation in addition to a legal one, then paywalls are vastly better than incredibly messy and problematic heuristic bot-detection. There's a huge payment problem for content anyway, so paywalls would mitigate or solve many other problems in addition to the AI-training-data one.

Reply on news.ycombinator.com