Comment by Kim_Bruning | Hacker News Reader

Comment by Kim_Bruning | Hacker News Reader

Spidering and Web scraping in and of itself is permitted by law (eu) or fair use (us). It is not considered illegal and many people do it for many different purposes. There are many common tools and libraries to help with this on linux, mac os and windows. It's even legal to keep the copies in a searchable database. [1]

What is not then permitted is to give other people copies, or publish them on your website, or pretend it's your own work etc...

When it comes to LLMs or image generation models, they don't keep any copies and they don't generate any copies either, so they consider themselves to be well in the clear. [2]

If you want to stop people scraping your stuff anyway, you can always use robots.txt , or put things up behind a login-wall.

Do consider the morality of what you are doing though. Personally I feel that published data should be scrape-able where practical.

[1] https://en.wikipedia.org/wiki/Authors_Guild%2C_Inc._v._Googl.... (you're even allowed to do this with physical books)

[2] https://www.uspto.gov/sites/default/files/documents/OpenAI_R... (With apologies for my crude summary of their actual arguments)

Publishing a derivative work of something you scraped is not permitted. The question is whether LLMs are derivative works, which has not been settled in courts yet.

Technically correct on the point that it has not been settled in courts yet.

For eg stable diffusion I actually dove a bit deeper on this:

The LAION dataset is -in itself- almost indisputably legal: It contains URLs of images, and tags describing those images[1]. To be sure: _It does not contain actual images_. People have made this mistake before, and LAION quite correctly reply that they don't have them. [2]

Stable diffusion's models are much smaller [3] than the LAION dataset which they use as their source. Again, the models do not contain images.

This is pretty suggestive that SD models should not be considered derivative works. Of course sometimes funny things happen in court, so we can't be sure until a court actually decides.

I don't currently run my own GPT or LLAMA, but those should have similar properties. (in fact, OpenAI appears to claim so.)

[1] https://laion.ai/blog/laion-5b/ (if you look for the files in any of the variations, you'll see it's in the order of terrabytes)

[2] https://www.vice.com/en/article/pkapb7/a-photographer-tried-...

[3] https://huggingface.co/stabilityai/stable-diffusion-xl-base-... : see under "files and versions", it's sd_xl_base_1.0.safetensors. File size is 6.9 GB

Not sure that the size of the model is relevant.

If you take a PNG of Mickey Mouse, and create a similar vector graphic that is a fraction of the size, that would still be derivative.

Thanks for the links! To me (merely an amateur observer), it is not perfectly obvious that the LAION dataset is legal, because I don't think "we don't have your original images anymore, we only kept the results of applying an algorithm to them" is a universal defense. If the algorithm is "resize to 640×480" you are probably still infringing, but if it's "calculate md5 checksum" you're probably not. I don't know where on the spectrum LLM training will fall, legally!

Heh, it does in fact calculate the MD5 checksum!

In total, the fields in the LAION dataset are:

  image_path
  caption
  NSFW
  similarity
  LICENSE
  url
  key
  status
  error_message
  width
  height
  original width
  original height
  exif
  md5

https://huggingface.co/datasets/laion/laion2b-multi-vit-l-14...

Note that LAION is a metadata dataset, it's not an image generation model itself. (But it has been used to train one)

I do not trust a court to weigh in properly on the deep technical nuances involved.

These kinds of things are not settled by arbitrary judges making off the cuff decisions.

They are settled by specialist technical judges with advice of dozens of specialists explaining the tech in detail.

Check out Google vs Oracle the lawsuit about whether an API is copyrightable for some examples of how the rulings are very much based on deep understanding.

The only downside is the process is slow and needs both a plaintiff and defendant willing to spend hundreds of thousands or millions of dollars to settle the case in court. It also takes many months of time.

last I heard about this ruling was that APIs are copyrightable, which is terrible for open source reimplementations of all stuff

More trying to focus on how the ruling wasn't naive as OP implies all rulings are by saying "I wouldn't trust a judge".

Alternatively they could mean "judges should stay away from X" but that is nonsensical given the courts being arbitrage...

we're talking about a law (copyright), who else can weight in besides the courts?

or are you suggesting abolishing copyright? because I'm all for it

This is an interesting topic that comes up time and time again. I'd like to understand, if we abolish copyright and I make a computer game, pour all my money and time in it, what's to stop EvilBigPublisher Co from simply taking it, rebranding it and killing my company with their might?

Or, if I came across a copy of Unreal Engine, what would stop me from using it and not paying Epic a dime?

1. Nothing stops them from doing the above now.

2. Unreal lawyers costing you more in legal fees than you can afford

Seeing as you are knowledgeable on the subject, may be you know if there is a licence, like creative commons, that will allow all the usual intelectual property uses except in AI training?

Or may be an old licence that is so strict that it will protect from AI shenanigans by default?

Reply on news.ycombinator.com