Hacker News Reader

Comment by jonathankoren | Hacker News Reader

I can’t help but think all this brouhaha over web scraping is just a rehash over the web scraping panic of the 1990s.

Put it behind a robots.txt or better yet a login.

This battle was fought and won. I’m not going back to the bad old days.

I want my website to be accessible to humans, and even indexed by search engines. I just don't want to be used as unpaid labour to train artificial intelligence.

A bit like I will gladly build a bike for someone, but object to the bike being sold around the corner.

I mostly agree with you, but as a counter-point: How would one precisely draw the lines between search engines and LLMs? One provides mostly airtight attributions, and the other is famous for hallucinating citations. But this doesn't sound like a distinction that can be used as the basis for law?

Or are you less concerned with their design and more concerned with their purpose? Providing citations is fine, but creating content is not?

One brings traffic to the original work, supporting its creators. The other takes uses the creators' labour without giving anything in return.

This was the same argument against search engines and in favor human curated hierarchies like original Yahoo and DMoz.

How do you Wikipedia being used for products period, let alone used as data products, since that’s also repurposed unpaid labor?

Wikipedia hosts people's hard work and makes it available to the world.

Search engines help connect people to the things they want. At least they mostly do.

AI inserts itself in the middle and strips the creators of credit, recognition or income.

Given that you used the term “AI”, you may not know that there’s a lot more to general web search engine beyond a simple inverted index. There are multiple models (ie “AI”) used to determine things such as query intent, term disambiguation, query classification, and that’s even before we get to results ranking, or relevant snippet detection. So now, with that knowledge, do you believe that Wikipedia, or even the web crawl itself (because that’s literally what we’re talking about, a web crawl) should be used for anything beyond term naïve term indexing?

Given the context, it's assumed that "AI" will not be used merely for providing better search results, but for keeping people on Google.com with "original" answers.

So just like the already existing info boxes, which I presume you have a moral problem with?

You can opt out of them. I'm fine with them, although they tend to be comically wrong.

If I want to read up more on this, could you throw some search terms at me that would help me find some more specific pieces of this conversation/debate around search engines' access to data? Any books, particular legal cases, prominent people or organisations engaged in this debate?

I feel the same way too, but I have to ask myself, why do I deserve a search engine to automatically index my content and then show it to others without my having to give anything in return? Let's say I actually wrote something of value, put an ad next to it, and it generated a hundred thousand organic search views. Those 100k views were basically given to me for free by a search engine. Do I owe them anything for that?

No of course you don't owe them anything. They opted in to doing this and didn't even consult you before doing so. They did this because they have found a way to monetize this, which is selling their own ads. On the other hand you just posted a blog post. You didn't opt in to anything related to search engines. You just posted something, that's it.

Yeah that's a good point. I didn't make a contract with them.

> or better yet a login.

That's what I've done along with a growing number of others -- but I'd much prefer to be able to make it available to the general public. I mourn that's not possible without also aiding the training of AI.

Reply on news.ycombinator.com