Google says it'll scrape everything online for AI | Hacker News Reader

Google says it'll scrape everything online for AI

gizmodo.com

64 pointsrntn85 comments

Well that's an interesting reversal. Back when I was a kid, websites would have terms of services that their users were expected to follow. Now, web scrapers have terms of... consumption?

Time to make a webring of Markov generated garbage to feed the bots...

Did Google say they aren’t respecting robots.txt files?

Does anyone respect robots.txt files?

Any history of Google not respecting it?

Google does not respect robots.txt if a page is linked to from other sites

https://developers.google.com/search/docs/crawling-indexing/...

That doesn't say the content but only that it is indexed.

Weird to see people get worked up about scraping the web for "AI" when search engine companies have been scraping the web for over 25 years.

99% of the time people's complaints about "permission" and "information accidentally made public" are the same arguments they have been using about search engines for years. Most of them have court cases even.

Scraping the web to reference your copyrighted material is completely different from scraping the web to use your copyrighted material without any kind of attribution or respect for your copyright.

If it is legal to do copyright laundering with AI, then copyright is dead. What's the point of writing a blog if people read it somewhere else, through an AI?

The problem is that a lot of the time the anti-AI people are complaining about the scraping itself. They say things "Downloading Copyright web content without permission" when thats what has been happening for years.

As if scraping was a new and unusual thing. Google and other have always been doing a lot more than just creating an index.

The opponents should attack the new stuff, not the other stuff that people have been doing for years.

> The problem is that a lot of the time the anti-AI people are complaining about the scraping itself.

Source? That makes no sense to me. The problem is what being done with the copyrighted material, not the fact that an algorithm loaded it.

Are you sure they complain about the scraping itself, or do you want them to complain about that, so that you can say that they make no sense?

OpenAI Faces $3B Lawsuit Over Alleged Private Data Theft

"The defendants are alleged to have conducted widespread web-scraping campaigns, violating various platforms' terms of service as well as state and federal privacy laws..."

https://www.iotworldtoday.com/security/openai-faces-3b-lawsu...

The thing with the law, is that you use whatever you can to win the argument.

But reasonably, I would assume that the humans who see a problem with AI are not bothered by the fact that some text appears in some memory somewhere. They are bothered by the use of that text, which they believe goes against their copyright.

It's not that web scraping has existed since basically the beginning of the WWW, but that Google has a hugely, massively and extremely unfunair advantage over OpenAI (who is getting reamed from all sides about the amount of scraping it did to train its models)

At this point, is there a way to protect your content from being thrown in the blender other than to not make it available publicly?

Perhaps something like AGPL but for text/documents?

Licenses alongside public content won't work because the reason people feel free to train is a legal theory that it does not require permission.

Do you really think that they don't train on AGPL code?

1. Win a lawsuit establishing that training AI is not fair use, or

2. Put it behind a click-wrap agreement that provides that it is provided to the user in exchange for an agreement not to use it to train AI.

3. For images, using on of the AI-poisoning techniques and publicizing that fact may encourage people to exclude your content from training sets; OTOH, it may backfire, especially against people looking specifically to defeat the AI poisoning technique in use.

Otherwise, those operating under the theory that training AI is Fair Use will have no reason not you use the content for training AI.

> Put it behind a click-wrap agreement that provides that it is provided to the user in exchange for an agreement not to use it to train AI.

Such a thing is only as good as your ability to enforce it, though. Are you in a financial situation that allows you to afford to sue Google? I'm not.

Set a price list for this type of usage, then pick a jurisdiction friendly to your cause of action.

Your ability to pick and choose jurisdictions is pretty limited (at least in the US). But ignoring that...

Having a price list doesn't change the fact that if someone ignores your demands, your only recourse is to sue them. If they have a bigger warchest than you, and are sufficiently motivated, they can drag the whole thing out until you run out of money and can't pursue the lawsuit anymore.

This is why such an approach is of limited value in terms of protecting yourself against companies that are sitting on a ton of money.

In memory of Don Lancaster, he made a point about this sort of thing: many companies would rather spend $100,000 on legal fees than stoop to paying you a $10,000 fee or royalty.

A small jurisdiction with dislike of USA hegemony may be a start, like .ch or .se. Somewhere with a Google presence and where you host your website.

I was rather explicit that the list was designed for dealing with “those operating under the theory that training AI is Fair Use”.

If your threat model is “those only concerned with what they can be forcibly compelled not to do, after evaluating the capacities of each content supplier to resist”, things are different.

Which category Google falls into on this is a debate I’m not interested in engaging in.

A variation on 1 is to get laws passed clarifying when it isn't fair use (in a way that allows you to exclude your work).

I've been looking into this pretty closely, and I've yet to find a good solution. The best approach I can find is to set up webserver rules to block connections coming from specific ip addresses, but that's just a whack-a-mole game I can't win.

So, for now, all of my public websites are no longer publicly available until/unless I can find a solution.

That's exactly the risk: if you don't want AIs to steal your work, don't make it public anymore.

To be honest if it's publicly available, I'm sure the odds that a human looking at your content and copying it vastly exceeds the odds of an AI utilizing it in its entirety. These are billion parameter models.

Depends on how niche what you do is relative to the quantity of information available on it.

If the person is accessing public information, they can plagiarize your content directly any way if that's the case. They could even think it up themselves, it happens all the time.

Ultimately the decision to check they're not utilizing copyright material rests with the human commercializing it.

A human copying your copyrighted material breaks the law. Doesn't mean they can't do it, but at least it's illegal.

And that's the crux of the matter. Specific words are input into the algorithm, to conjure your code from a probabilistic model. Use a random sentence generator enough and at some point it'll plagiarise. It's how the human uses it's output is what is important at the end of the day.

If you believe that humans have an infinite memory and can optimize to be just on the right side of the plagiarism line, then I suppose it is the same thing.

Now my opinion is that humans are not machines, and therefore it makes sense to not apply the same laws. To me, it seems natural to think that a human will never remember every single word of a book they read. Whereas a machine can totally do it.

> It's how the human uses it's output is what is important

I would take it from the other direction: the AI should be able to prove that it "read" the words like a human (i.e. with a "bad" memory and everything that goes with it).

A computer can remember to do that only if it's programmed to do so.

These are probabilistic models,it's basically a large algorithm. They're not copy pasting a database verbatim. They don't have access to it, only a list of parameters (sort of the algorithm weightings) and the list of tokens (translated words).

With OpenAI's API you can even see the certainty/probability associated with the next word it quotes. We know for definite whether a model is doing this by looking at it's source code. Humans do also come up with the same ideas from time to time, without any external input.

Wait, so you're saying that you can go look at the billions of trained parameters, and tell me for sure that my blog post is not "encoded" there (for any definition of "encoded" that would allow the model to generate text that would reasonably be considered as plagiarism)?

> A computer can remember to do that only if it's programmed to do so.

Not sure what your point is there. So your belief is that if the programmer did not explicitly put a specific intent into the code, then the computer cannot do anything consistent with that intent? Like nothing unexpected can ever happen with computers, because they are strictly limited to what the programmer explicitly wanted them to do?

A random word generator given enough time could do that.

Given these models have billions (hundreds of billions) of parameters, the odds of it exactly copying your code is less than 100%.

Academics when publishing run through plagiarism checkers as it happens even if they're 100% sure they haven't consulted other thesis'. (Usually 20%-30% is acceptable).

Some coders run their code through checkers to make sure it's not copyrighted. The onus similarly would be on whoever is using the LLM output to check it before monetizing it. It's just a tool after all.

I am unconcerned with humans reading, copying and using my content - that's the whole point!

What I am concerned is big conglomerates munching on it and reselling it at a high price.

They're selling probabilistic models, not your content (well let's hope). More akin to a search engine database.

> They're selling probabilistic models, not your content (well let's hope).

A distinction without a difference.

> More akin to a search engine database.

The point of a search engine is to direct you towards the content. The point of LLMs is to keep you within the walls of the LLM.

What's the point of having it public for humans but not for machines?

In principal I think it’s less about humans vs machines and more about individuals vs corporations. It’s one thing to want to do contribute to the global knowledge base. It’s another thing entirely to monetize that knowledge wholesale and that’s what AI companies seek.

> It’s one thing to want to do contribute to the global knowledge base. It’s another thing entirely to monetize that knowledge wholesale

You mean like a search engine scraping content and putting ads by results?

Exactly. There are existing mechanisms[1] to tell Google that it’s not allowed to scrape/index your site.

1. https://developers.google.com/search/docs/crawling-indexing/...

"Public for humans" is not unconstrained. Permissive open source licences require attribution, copyleft licences have all sorts of requirements.

If AI could honor those licences (and all the others), I wouldn't see a problem. But they can't.

I object to the use of my material for helping to train AI. Humans are far less likely to be using it for that purpose, if only because you can't do it at scale with humans.

I think of making something like Wikipedia where I write loads of useful stuff an throw in a handful of dick pics - not mine - but you get the general idea.

There are also more sophisticated attacks like even single pixel attacks that one can place in his content.

I expect if I put enough counter AI attacks in my websites media/text they will notice that and put me on their exclusion list.

Well they are first to pull asshole move so won’t feel guilty.

I rather like how you're thinking here. I think I need to look into how I can throw the biggest monkey wrench possible into the data AI scrapers get from my websites. Make the bastards pay.

Make it commercially unpalatable. Nobody will touch 4chan.

Have different service levels. Authenticated users get the real thing (rate limited), anyone else gets a dynamic Markov-generated version with subtle errors deliberately introduced.

No. The US Supreme Court has ruled that a website cannot use TOS to prevent other companies from scraping publicly available data, even if they're doing so to directly harm the people who posted that data (citation below).

Basically, a company called HiQ was scraping public LinkedIn data to sell a service where they would inform companies if their employees were looking for a job on LinkedIn -- because LinkedIn will hide info from your employer for obvious reasons. LinkedIn sent cease-and-desist, then later sued by lost.

https://www.fbm.com/publications/what-recent-rulings-in-hiq-...

The ruling states you can't use the CFAA to prevent people from scraping. It says nothing of copyright, and indeed you can't just ignore licenses when scraping, otherwise every public repository on GitHub would be a free-for-all.

If we don't make this illegal, then I believe copyright is dead. If you don't want your stuff to belong to the AI that wins the race, just don't publish it anymore.

Try scraping at scale from them, I’m sure they would graciously take it as a fact of life in the open internet. You wouldn’t hear a word from their lawyers, no sir!

Serpapi.com has been around for several years now, in fact if you pay extra they agree to indemnify you against lawsuits by Google regarding scraping (as far as I can tell none have happened yet).

I'm not mad about it. Have at. Certainly AI can only benefit from my deep insights and hard-earned wisdom. I can see why some people are mad, though.

Are you writing content (like a blog)? Would you be fine if another website copied your blog without giving attribution, and people read from that one instead? Would you still write blog posts if people read a modified version of them through some AI system without ever knowing about what you wrote?

Are you writing open source code? Would you still make it open source if people never saw it as "your" code, but instead paid an AI company to access your knowledge?

Are you getting those strange internet points by answering StackOverflow questions? Will you still do it if people stop seeing the answers you redacted (and voting for them)?

If it goes there, I will stop blogging, stackoverflow and open sourcing. And everything else that I do for free, just for the satisfaction of getting a "thank you" from a human from time to time. If AI wants to make me disappear, I will disappear.

I wouldn't mind if this encouraged us to get offline and invest more time and attention in our local communities. I'm ready for the Star Trek future where we don't have jobs, and we do things for their inherent merit.

Would I get mad if someone read my blog, learned things, and then wrote their own blog using some of these ideas? No.

But that someone won't read your blog anymore, they will read a modified version on some AI frontend. Maybe modified in a way that alters your meaning. So not only you won't get any kind of attribution ("look how clever the AI is, it just learned the world and explained this to me"), but possibly your point won't get through the way you intended it.

What some LLM spits out is not my post anymore. My post is still correct and as intended. Someone could read my post and retell their own version at the pub, that fine, it's their version.

But the LLM won't say "please be careful, this is a modified version of Gigachad's post that you can read at <link>", will it?

Don't you see a difference between 1) a human reading your blog post, understanding it, making their own opinion, and then sharing it under their name and 2) a machine taking your blog post as an input and distributing a modified version of it?

Let's take two examples:

First example: I create a blog that copy-pastes your posts to the word, but I don't give any kind of attribution, and I get money from it. Would you be fine with that, or would you consider that I should not be able to do it, because I am basically selling your copyrighted work without authorization?

Second example: I do exactly the same as above, but I have a small script that will replace some words with synonyms. Is it now better than the previous example?

Where do you put the limit? If the script is so good that you can't recognize your post anymore, then you find it okay? But that's still an automated script that just duplicates your blog and benefits from it, right?

The pirate in me dreams of the conversations I could have with libgen.ai

GPT-3 was trained on it, so just use ChatGPT

Until the maintainers of libgen are fed up from the fact that people think "it's ChatGPT" and don't know about them, at which point they may just stop the project.

Not scraping the public web is a competitive disadvantage, since OpenAI and others are definitely going to do it even if you don't. Unless there are new laws regulating it or some court setting a precedent under existing copyright laws, you can just expect all publicly viewable data online to be constantly training AI models.

Which is why we need laws, otherwise copyright is dead.

How's that Google's privacy policy? It even applies to entities that have no relationship with Google. It's about those other entities' privacy policies (Google might ignore them) or a statement about Google's business practices.

I'm thinking, what if someone would add in their blog's terms of use something like "the content on this blog is not for machine learning purposes"? Could they later sue Google if their site gets scraped by Google for LLM training?

If those terms of use were machine readable, this would have been equivalent to something like robots.txt, but for AI.

https://news.ycombinator.com/item?id=34324208

https://news.ycombinator.com/item?id=35886288

A contract requires an agreement between two parties. I don't see how writing "the content on this blog is not for machine learning purposes" alone shows Google has agreed to your terms.

Next you're gonna tell me the copy-paste Facebook statuses that say Facebook doesn't have rights to your photos don't do anything!

The US Supreme Court has already said no, see the case of LinkedIn vs. HiQ.

https://www.fbm.com/publications/what-recent-rulings-in-hiq-...

from new T&C

For example, we may collect information that’s publicly available online or from other public sources to help train Google’s languageAI models and build *products* and features like Google Translate, Bard, and Cloud AI capabilities. Or, if your business’s information appears on a website, we may index and display it on Google services.

https://policies.google.com/privacy/archive/20221215-2023070...

Are they using the contents of GMail email messages ?

That is not public, I guess they are not. I mean not for the public-facing AIs. Obviously they use them internally.

> I guess they are not

For situations like this precision matters and guesses don't count.

Well, I would think that this is not yet solved. If AIs are allowed to use copyrighted open source code (without honoring the license terms) like they are currently doing, why couldn't they use copyrighted proprietary code, like the leaked sources of Windows?

And if they can use leaked proprietary code, why couldn't they use basically everything they have access to, like your emails?

In my opinion, it should all be forbidden, but you will find plenty of voices online that think AI should be able to (ab)use open source code, so...

> In my opinion, it should all be forbidden

We're fully in agreement there.

I remember the first time I really understood Google was using my "private" comms to sling ads at me. The writing on the wall seemed clear then. I just didn't know what the future endpoints were.

I recently started scraping internet for my own work. Unfortunately the sad part of using public data for AI is there is no references to the original work.

Does this include google photos and google docs?

Neither of those are end to end encrypted, so yes.

They have more than enough data for good models. Training good models that work is the hard part it seems.

I consider it like using photos in public. It's legal. People are being so weird about this.

Artists and writers are supposed to be ok with , why not everybody else?

Simply, if you provide a thing for free, expect people to take it and use it for free. Artists, writers, programmers, and everyone else included.

If you do not want your “intellectual property” taken and used by others, don’t give it away for free.

Artists are expected to have online portfolio's of their work easily accessible for prospective clients and selection committees for any kind of shows or programs that they apply for. It is considered a mark of professionalism. Jpeg portfolios are too large to send like a resume. The very tool artists use to get work is being used to take work away from them.

Reply on news.ycombinator.com