Hacker News Reader

Comment by layer8 | Hacker News Reader

SOTA models cost hundreds of millions to train. Did creating the contents of the text corpus they were trained on really cost an equivalent of 20x as much (~10 billions)? I honestly don’t know, but I could imagine it having been significantly less.

This isn’t meant as a moral argument, just musing about the relative cost comparison.

View on HN

If you look at movies alone that would easily surpass 10s of billions. The cost of most books is probably more nebulous, but books, research, and more all have time and money spent to create them. I would guess the corpus of all media from the 20th century on would be minimally in the hundreds of billions of dollars.

LLMs aren’t trained on movies, though.

Image/video models are, but those weren’t the topic.

Given they ingested basically the whole internet and then some, you cannot possibly be serious when you mean it's worth less than 10 billions.

The totality of the content on internet is worth several orders of magnitude more.

The argument wasn’t about how much it’s worth, but about how much it cost to create. These are very different things.

do you also count eg published results of very expensive physics experiments? because once the costs of things like these are taken into account, we are way over 10 billions.

Why? Are we doing labor theory of value now?

I would argue that producing the complete written corpus on which they at least intend to train (even if some is still out of reach) cost literally everything to produce.

And the monetary cost doesn't even register when weighed against the blood, sweat and tears that went into capturing the authentic experiences of real human beings, whose honest expressions are now at least in some cases getting hoovered up, ingested, and then destroyed for all eternity, for fear that this specific work is the rounding error that might give an equally immoral competitor the edge in the bicycle-riding flamingo race that is currently consuming an absurd amount of the world's creativity and attention.

Simple math:

A training set of 15 trillion tokens is 10 trillion words.

A penny a word is cheaper than the cheapest beginner freelance writer.

That makes a training set of 10 trillion words cost $100B.

Lots of assumptions there for sure, but we're certainly in the ballpark you are describing.

How much did you get paid to write this?

I asked Claude to estimate the cumulative salaries of US only journalists over the last hundred years:

$500B for all kinds including TV and online

$300B for newsrooms including all staff

$140B for newsroom reporters only

So yeah, I think the price of the information ingested is way higher than training costs

Yes, Easily, and by multiple orders of magnitude.

I suspect so. It is a lot of data. You're looking at essentially all publicly available (and some non public) intellectual work.

Yes, duh! Human output across the millennia is worth much more than whatever is being invested in frontier labs.

How is this even a question.

They are solving unsolved math problems right now, so probably soon or very soon their output will be more valuable than all human recorded knowledge.

Pre vaccination smallpox killed hundreds of millions of people just in the twentieth century [0], the knowledge that allowed for the creation of just that vaccine is worth hundreds trillions of dollars in humans lives, let alone all of `the knowledge and experiences those people were involved in.

The knowledge that created the Haber-Bosch process [1] helps to sustain the majority of the world's populous, add another five hundred trillion dollars for that just to start with.

The creation of the printing press and all written information that allowed it to be built provided dissemination of knowledge beyond the ultra wealthy and is worth a non-finite amount of money.

LLM's are cool math, but they are less than a rounding error in comparison to even the tiniest sliver of human knowledge and technological output.

[0] https://pubmed.ncbi.nlm.nih.gov/35143880/ [1] https://cen.acs.org/food/agriculture/The-industrialization-H...

What do you think mathematicians were doing for centuries before LLMs?

And also, humans have been doing a lot more work than just mathematics...

this is the sort of brain rot thought that you have in a dorm room the day you are introduced to Econ

“bro like, what if we could price the sum total of human knowledge? That wouldn’t be that much, right?”

Reply on news.ycombinator.com