Publishers want billions, not millions, from AI
semafor.com
semafor.com
LLMs are a genuinely exciting technology, but I am worried that a part of what they enable will turn out to be a sort of "content laundering." If you train your AI on a bunch of someone elses work, you can make it produce something very simiilar. You can extract whole aspects of an artist or writer's style and replicate their ideas much more easily now, and because it is generated inside this black box you have a sort of deniability when it comes to accusations of copying.
As a blogger of 10+ years this has been very interesting to realize. I can use ChatGPT to write really good first drafts because “in the style of Swizec Teller” works as a prompt. The results are way better than the usual generic corporate drone style of output.
BUT! And this is an important but. The insights it produces remain at that generic corporate drivel level. Even with a full list of bullet points of cool insights, an LLM takes those and averages them out into a big pile of meh. Every sharp insight gets dulled by “AI protections”, every hot take is two-sided to death, etc.
Despite superficially following my style, the AI seems unable to catch your attention. Everything it writes is a little boring, a little too long, a dash tiring to read. The spark is missing.
Also my style has evolved since 2021 and that’s missing.
This is a OpenAI thing.
Run a "uncensored" llama finetune, like airoboros 70b. They are as hot and spicy as you want them to be.
EDIT: Actually someone is hosting 65b here, so you can test it and see what I mean: https://lite.koboldai.net/
Note that you have to get the instruct prompting syntax right.
Relevant Zizek: https://www.newsweek.com/slavoj-zizek-self-plagiarized-new-y...
That depends. For me the fun part is in producing the novel insight. But nobody wants to read 5 bullet points and they wouldn't get it either.
Turning those 5 bullet points into a narrative that guides readers towards understanding, that part feels like work. If AI can create the skeleton that I then edit into good writing, that feels like a win.
The writing process for me goes something like this: 1) Read a bunch of books, 2) Percolate/simmer/gestate for weeks, 3) Observe reality and seek anecdotes, 4) Simmer some more, 5) Wake up one day with 4 insightful bullet points, 6) Turn that into 600+ words (this part feels like work)
To give another "scale makes all the difference" analogy. Very early on when Google Maps Street View was originally released, there was debate about whether it was OK to show individuals faces. In the US at least, the argument went something like "People are out in public, they should have no expectation of privacy. If it's legal for a person to take a picture of someone else on the street (which it is in the US), why should Street View have any privacy concerns?"
The difference is that while I may expect that other people may see and even take a picture of me if I'm outside, that's different from making my picture searchable, geolocated, instantly available to billions of people across the world and online forever. And I think it's totally rational to think these 2 different situations require different approaches.
Lots of our previously common, intuitive notions of what is OK can change greatly when large scale automation enters the mix.
Mechanizing that process so that it can be wielded at scale and automated is a pretty significant "change to the contract", and worth exploring what compensation is fair under these new circumstances.
The amount of times I've gotten into a cab in London before Uber and the cabbie was like: "Cash only". What's that machine for then? "Credit card machine down". Yea right...
I absolutely hate what Uber did with 'contractors' but I cannot deny the fact that I like being able to travel for work without having to worry about whether the cab has a credit card machine or not.
That entirely depends on where you are. Where I am, ride-share services were never better than taxis in any way except for price.
I've started seeing content sites that are obviously generated content looking at it... mostly in terms of recipe content for lower carb, or sugar free... some list ingredients, but no measurements, others list directions that don't match ingredients. TBH, these kinds of activities make the internet less useful overall.
Between the above and sock puppet accounts, I can't help but think it may be a time to return to self-hosted systems like the BBSes of the 80's and early 90's. I know some still exist, but they're mostly around those that were into the tech, or otherwise into the older turn based text games or artwork. I know there's various systems from discorse to lemmy that are a bit more modern. I can't help but think something closer to a single server version of Facebook Groups +_Chat might go a long way... not even federated, but single-interrest groups that aren't beholden to a larger org (reddit, etc).
Part of the popularity of these sort of things stems entirely from how much money is reportedly up for grabs. If you tell people they can do this and bring in 2k a month on the side, they're all in. Likewise, if we had enough funds to not have to worry, these sort of things would rarely be used as there's no real joy in the pursuit of it. My theory on why AI is accelerating garbage content much faster than any other potential benefits.
Even the highlight of recipes is notable, recipe blogs are great earners due to the stability of ads and output of content. This becomes a target for AI (when I say AI here I mean people consuming AI API) due to the high ad value target. That way even if they do eventually drop, get found out, reported to google, all of that, they would have still made enough to do it again.
I couldn't agree more. We need to stop using other people's computers to host our stuff. (Or, at least we need to rent the machines ourselves). These days, the BitTorrent protocol is popular enough that you could likely host the equivalent of a YouTube channel for very little actual bandwidth cost.
Text, forums, etc... without media, are almost free until you scale into millions of users.
You don't have to keep sharing though, once you've downloaded it. It's voluntary.
Not really what I meant.
> instead the kept bleeding their content creators of all they have
But yes, that. And their customers too.
> “Search was designed to find the best of the internet”
Says a publisher who clearly hasn't used search in the past 10 years.
The thing is if this upsets you on some gut level, this is just a sign for how valuable information to train on is. May be billions is too much, but hundreds of millions? If your immediate thought is "this must be stopped, this isn't fair" it is a signal of how reliant AI at all is on the data it trains. An untrained nn is useless, it's only useful if it has information to train on. It's not like value creation is zero sum but it certainly doesn't come from zero, and the "something" it comes from is definitely more of the data you train on than the training method you choose.
Without formal agreements with content owners, OpenAI is just another scraper. One can easily imagine that content cooperatives may ultimately own the GenAI space.
"What I want to talk about is whether we will get royalties from the elimination of all humans"
Let’s say for example that they trained on the content of the entire archive of the New York Times… isn’t it safe to say they’d have purchased a license for that content from NYT?
Wouldn’t “commercial use” cover this?
Seems like the publishers just want a piece of the money because they want a piece of the money.
The legal status of copyright in this context is unclear, because the laws were written without imitation machines in mind. Presumably they blew right on by "is this right?" and went for "can we get away with this and pay less in fines than we'd make in profits?" As long as there's a 51% chance of that being the case, they saw it as worth a go.
The problem is that companies see that they undersold their content relative to the value it’s providing LLMs, and now want to renegotiate past and future deals with AI companies. Exactly as you said: they just want a bigger slice of pie.
Ha.
There is a difficult to describe culture in the ML community, from the top researchers to casual tinkerers... Everything is bleeding edge and moving at blinding speed. Some devs don't even stop to think about things like licensing or easy reproducibility/packaging, because they are too busy moving onto the next thing.
This thread on how a popular repo was unlicensed and violating other licenses for months is a good example: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issu...
OpenAI comes from this culture, even if they are a more commercial company now.
More broadly, it's not a settled question of what license you need. The article presents a perspective that LLMs "would turn a Food & Wine review into a simple text recommendation of a bottle of Malbec, without attribution." I doubt that anyone has a license to republish the entire archive of the New York Times without attribution.
At the other extreme, I can "train" myself on newspapers I find discarded on the subway with no license at all. I don't need to cite my sources when I state an opinion based on everything I've ever read, seen, or heard. Is that a fair analogy to apply to LLMs?
Well, how much harm would you, a single person who also needs to earn a living and probably wishes to write your own original content, be for the original author?
Compare that to a datacenter full of servers ready to output millions of articles per minute in the exact style of the original author for a per-article cost close to zero.
Sure, but of course, we all know the actual value came from the writers.
Why not a different rent seeking model.. For pharma companies? If your product cures a patient, all other pharma companies own you a percentage from everything sold ever after.
Now that one company closed it off, is trying to regulate things, and is profiting from it: Let the lawyers win with fees.
That nightmare scenario, for Levin,
would turn a Food & Wine review into
a simple text recommendation of a
bottle of Malbec.
Nobody should be allowed to bend the law and extort money from society for writing wine reviews.
Mankind had this discussion before. When Darwin published his theory about evolution. He faced a lot of hatred because it made humans less special.
Now we go through the same dance again. This time, intelligence in silicon makes humans less special.
I understand that people like to anthropomorphize things. That's harmless fun when people are imagining tree spirits or thunder gods or whatever. But just because fancy autocomplete produces semi-intelligible text does not mean there is an intelligence behind it. I believe we will eventually be able to create synthetic minds. But our understanding of actual minds is so poor that it's going to be quite a while before we manage.
More than 200 years ago, Mary Shelley told a story of a scientist creating life via the then-new technology of electricity. Since the we keep re-telling the story with the latest technology. It's a good story, and like all good art helps us understand what it means to be human. Which is why I think it's especially bad to use those stories to muddy the many differences between actual humans and something that produces statistically plausible generated text.
If nothing - where is that intelligence you are talking about? That intelligence that only humans have?
Virtually everything you "know" is because of current and past humans. Have you been paying everyone for all of that?
If you want to be a great author and you read books by great writers.. then become a successful writer, are you paying all those authors for training you?
I realize there is a difference with being able to absorb bulk data and all that- however, at the core the argument is still the same.
Do you regularly copy protected software and claim it as your own creation?
I’ve seen this argument every time someone states that these for profit ai companies should not have complete access to a all of human knowledge and I just don’t see how we’re living in the same world.
One way to tell is that we think protecting human creators is so important that the US Constitution says Congress has the power to "To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries".
So yes, we absolutely try to pay all those authors. And we also have a variety of cultural mechanisms to reward them in other ways, from honorary doctorates to literary prizes to genius grants to homages. Not to mention a widespread hatred of plagiarism that goes well beyond what copyright covers.
None of that is respected by the current crop of text generators. They chose to release these things knowing of these lacks. And that lack of respect is being richly returned, as it should be.
Teachers are paid, tuition to schools is paid, feedback from tutors is paid, textbooks are paid for, copies of books are paid for, movies and TV are paid for. The experience you learn on the job is paid for via yours or your coworker’s salary.
I get the argument that it’s not copying it’s learning, but it’s also very different than a human learning by observation.
And if learning weights is okay, is it okay to train an AI with the structure designed to reproduce Shrek, then sell the AI? I think ShrekAI would be illegal in most countries. It’s a blurry question that we need to think about.
https://www.americanbar.org/groups/science_technology/public...
In order to know that many reviews recommend Malbec, it needs to be taught this fact specifically, or infer it from context.
Do more Wikipedia pages mention the word "blue" or do more Wikipedia pages mention the word "Accismus"?
ChatGPT:
To determine whether more Wikipedia pages mention the word "blue" or the word "Accismus," we can use a simple search on the Wikipedia website.
Searching for the term "blue" on Wikipedia yields a large number of results, as the color blue is a common topic across various fields. On the other hand, "Accismus" is a relatively obscure term, and it is less likely to be mentioned in a significant number of Wikipedia pages.
Therefore, it is highly probable that more Wikipedia pages mention the word "blue" compared to the word "Accismus."
But the training process is not a library it can draw upon or answer questions about intrinsically.
So how do you propose to pay for someone to write wine reviews?
But why ask for billions when you can have ... millions?
They are in no position to make demands. The current data that has been scraped for AI models is probably good enough to be able to generate synthetic data.
Even if it's not, what are publishers going to do about people scraping their content? Absolutely nothing.
Trying to force this issue feels like a good way to be disintermediated quicker.
Generating overpriced intangible goods, the only value of which is novelty is almost necessary in capitalism to keep some form of constant demand. The alternative is continuously. burning something, like we do with oil.
I wonder how we are going to keep all that money in circulation, once we do not need a trillion dollar industry of entertainment anymore.