Comment by vessenes | Hacker News Reader

Comment by vessenes | Hacker News Reader

This is a good essay, and makes me hopeful.

I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.

For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.

But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.

Everyone in the "if not us, they will" race is brainwashed into thinking they belong to this or that party, while in fact collectively comprising the same entity that pushes forward all the atrocities known to man.

No. These parties are composed of people who most definitely think this way, and therefore will have distinct goals and interests when presented with opportunities. That’s reality quite aside from how a game theorist assesses the situation.

I cannot tell what negative-sum outcomes you consider possible. Do you believe AI can drive humans extinct? How many of Zvi Mowshowitz's Three AI Pills would you say you've taken?

https://thezvi.substack.com/p/the-three-ai-pills

I’m like a 2(.5?) there - I don’t think ASI will care about my kids better than I will for some definitions of better, for instance, and I feel very fuzzy and vague about what actual differences in qualia between me and ASI would yield in the wild.

I’m not a doomer, although I don’t think doomers are dumb, just wrong. I think you should design your systems around the possibility that people who disagree with you are correct , hence my nod to negative sum. If you have more than 30 years to live, I’d personally rep to the most likely outcomes being very positive. With a lot of disruption in the middle.

> if you have any strategic adversaries whatsoever you MUST NOT slow.

What if the most dangerous strategic adversary you have is the one you are building?

What if this is true mid or long term but by not participating to the AI race one gets poor or killed in the short term? The only way out would be that all parties agree to stop. There are previous examples (e.g. nuclear proliferation treaties) but it gets hard to do it with hundreds or thousands of parties.

I don't think it requires the agreement of that many parties. How many organizations/physical sites can create chips capable of training and running frontier models? That is your bottleneck. It is equivalent to targeting uranium enichment in nuclear arms control.

Although many share your mindset, I’m glad there are also many that don’t. Otherwise we’d still have countries in a race to keep building up their nuclear weapons for the same exact reasons you just described.

The situations aren’t equivalent - luckily in my opinion because the stakes with nuclear are much higher. von Neumann constructed a multinational game theory approach appropriate for weapons. AGI is a much harder problem to corral because there are so many benefits beyond just blowing up cities. But it’s also a much better thing to have for these very same reasons.

Similarly there have been few positive externalities from nuclear industry, making it easier to make the case to wind down research. This same set of concerns in biotech is much harder to get compliance with, precisely for this reason.

Anyway I’m especially wary of over analogizing to nuclear era concepts: I think they’re a trap.

In your opinion, what are the top 2 positives and top 2 negatives of humanity inventing AGI?

slowing can also make sense if you know you're running full force into a bomb or a wall even if other are close behind.

If you must not slow, why did we slow down making nukes? Seems that sometimes, eventually the rat race goes on long enough where all the players no longer care to play into the farce like their predecessors who passionately beat that drum.

Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.

> Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.

Do you post this comment on every single blogpost with a corporate domain? Why or why not?

i have a new strategy idea for using capitlaism itself to slow down the pace of AI development by slowing down the data accumulation wall

https://jperla.com/blog/the-data-tax

hm- does the model that wrote this know that labs already pay for training data- that stuff scraped from the Internet is not particularly where today's capability gains come from?

They’ve settled some lawsuits and have a few licensing deals, IMHO they are not free from the accusations of pirating.

And look, I’ve pirated material in a past life, I was all about information wants to be free, but I’ve learned something about consent since then and try not to ignore the contract that creators offer when they publish something: you buy my book, and do whatever you want with it on the second hand market. Buy my book second hand that’s fine. But don’t go downloading every book that’s ever been scanned to create a service that destroys writers’ ability to make a living and act like you’re doing us all a favor.

The point is, the big improvements we’re seeing nowadays are coming from RL, not from scraping the internet.

the point isn’t scraping it’s taking your data and enterprises data

https://trustedrouter.com/blog/they-are-still-training-on-yo...

Where are you getting your information from? From all I've seen the RL gives an incremental improvement, most of the capability increase comes from new model architectures (eg the jump from opus to fable is greater than the jump from opus 4.5 to 4.8)

they pay for some data but they take all of the stuff you’re throwing in too; that’s why i propose forcing it since they’re already used to paying for data just increase the cost even further

https://trustedrouter.com/blog/they-are-still-training-on-yo...

This is a bad essay, or rather it’s a marketing fluff piece; it’s certainly not any kind of policy paper, research paper, or even an essay. I am concerned that we (meaning, we in the tech industry) tend to take this type of writing for more than that.

Reply on news.ycombinator.com