duvenaud

Because I think being permanently disempowered is likely to lead to extinction eventually. I also think there are other plausible, faster routes to extinction.

I'm not totally sure, but humans are better than our ancestors at basically every important job. It would be surprising if human brains were the pinnacle of computational power, and even if they were, digital brains could be run faster + copied.

I think it's very unlikely that any of those would lead to human extinction, especially since most of those take decades to unfold, and would still leave large parts of the earth habitable.

Toby Ord wrote a book trying to estimate each of these risks, and put AI on top: https://www.amazon.ca/Precipice-Existential-Risk-Future-Huma...

I'm confused. You agree that we're surround by naturally-arising, self-organizing agents, both biological and institutional. People are constantly experimenting with agentic AIs of all kinds. There are tons of theoretical characterizations of agency and how it's a stable equilibrium. I'm not sure what you're hoping for if none of these are reasons to even worry.

Good point. Although in some ways it's a lower bar, since agents that can control organizations can delegate most of the difficult tasks.

Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.

Sorry, my gun analogy was unclear. I meant that, just because some agents on a planet have an ability, doesn't mean that everyone on that planet benefits.

I agree that we probably won't see human extinction before robotics gets much better, and that robot factories will require lots of infrastructure. But I claim that robotics + automated infrastructure will eventually get good enough that they don't need humans in the loop. In the meantime, humans can still become mostly disempowered in the same way that e.g. North Koreans citizens are.

Again I agree that this all might be a ways away, but I'm trying to reason about what the stable equilibria of the future are, not about what current capabilities are.

This is a great point. But I'd say that capable entities have a habit of turning themselves into agents. A great example is totalitarian governments. Even if every single citizen hates the regime, they're still forced to support it.

You could similarly ask: Why would we ever build a government or institution that cared more about its own self-preservation than its original mission? The answer is: Natural selection favors the self-interested, even if they don't have genes.

I guess so, but the difference between "humans are extinct" and "a small population of powerless humans survive in the margins as long as they don't cause trouble" seems pretty small to me. Most non-human primates are in a situation somewhere between these two.

If you look at any of the writing on AI risk longer than one sentence, it usually hedges to include permanent human disempowerment as similar risk.

Yes, I agree real life is fuzzy, I just chose chess as an example because it's unambiguous that machines dominate humans in that domain.

As for being competitive with humans: Again, how about running a scan of a human brain, but faster? I'm not claiming we're close to this, but I'm claiming that such a machine (and less-capable ones along the way) are so valuable that we are almost certain to create them.

Again, are you claiming that it's impossible for a machine to invent anything that a human could? Right now a large chunk of humanity's top talent and capital are working on exactly this problem.

As for your second point, human cities also require a lot of infrastructure to keep running - I'm not sure what you're arguing here.

As for your third point - would a horse or chimpanzee feel that "we" were unbeatable in physical fights, because "we" now have guns?

I see what you mean - for a while, the best chess was played by humans aided by chess engines. But that era has passed, and now having a human trying to aid the best chess engines just results in worse chess (or the same, if the human does nothing).

But whether there a few humans in the loop doesn't change the likely outcomes, if their actions are constrained by competition.

What abilities do humans have that AIs will never have?

AI Safety hasn't been divorced from tech companies, at least not from Deepmind, OpenAI, and Anthropic. They were all founded by people who said explicitly that AGI will probably mean the end of human civilization as we know it.

All three of them have also hired heavily from the academic AI safety researcher pool. Whether they ultimately make costly sacrifices in the name of safety remains to be seen (although Anthropic did this already when they delayed the release of Claude until after ChatGPT came out). But they're not exactly "watching from the sidelines", except for Google and Meta.

Here's one of my concrete worries: At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.

I agree we don't necessarily know the details of how to build such a system, but am pretty sure we will be able to eventually.

Well, Rees co-founded the Centre for the Study of Existential Risk, and has at least been thinking and writing about these issues for years now.

I signed the letter. At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.

Thanks for asking. I mean, my brain is just a machine, and eventually we'll make machines that can do everything brains can (even if it's just by scanning human brains). And once we build one that's about as capable as me, we can easily copy it.

I signed the letter. At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.

Yeah, I agree that there'd probably be ways around this patch such as the ones you suggest.

I agree, and this makes my proposed patch a weak solution. I was imagining that the specialness of the token would be reinforced during fine-tuning, but even that wouldn't provide any sort of guarantee.

Here's a potential patch for that particular issue: Use a special token for "AI Instruction" that is always stripped from user text before it's shown to the model.

Perhaps a better place to find algorithmic details is this related paper, also with Hinton as a co-author, which implements similar ideas in more standard networks:

Scaling Forward Gradient With Local Losses Mengye Ren, Simon Kornblith, Renjie Liao, Geoffrey Hinton https://arxiv.org/abs/2210.03310

and has code: https://github.com/google-research/google-research/tree/mast...

Thanks for the explanation, but I'm not sure how it relates to my comment. It sounds like you're saying regulations help because they force banks to keep reserves. I think this helps in the short term, but in the long run creates incentives to deceptively package assets into AAA ratings (as in 2008), and that things like FDIC also just create incentives to get too big to fail. So I stand by my statement.

I mean, banking was already hundreds of years old in the 2000s. So recent bubbles and crashes in traditional markets seem like evidence that regulation, at least the kind that ends up being implemented, isn't doing much to prevent very bad outcomes.

It is a parody.

>We also know that the diffrence between salaries of rank and file staff and C-suit executives have increased 15x in the past 20 years, without any corresponding increase in productivity

Seems like evidence against the claim that status plays a significant role in setting salaries, unless their status has also increased massively in that time.

Thanks for providing more context. However, the original claim I was reacting to was that "half the population ... [are] beneficiaries of affirmative action", i.e. that all men in college are beneficiaries of AA. I agree there are exceptions, but I stand by my claim that very few college-attending men benefit from AA for males.

> While I believe there is belief that Harvard is accepting C-students over asians A+ students I think the reality is much closer; they are rejecting students with perfect scores and accepting students who scored in the 98th percentile.

You don't need to speculate, the gap is more like 99th vs 93rd percentile:

"Asian-Americans admitted to Harvard earned an average SAT score of 767 across all sections. ... African-American admits an average of 704."

https://www.thecrimson.com/article/2018/10/22/asian-american...

As a STEM academic, I can say that I've never seen or heard of ongoing AA for men in any STEM program in any institution whose policies I'm acquainted with. So it seems to me to be misleading to claim, or at least hard to believe, that very many men in college have been beneficiaries of AA for men.

Thanks for elaborating. I'm still not convinced about the strengths of claims that can be made in this instance, but I like the analogy to the 4-minute mile. However, I think that people didn't determine that a 4-minute mile was possible by using their knowledge of human anatomy. But I concede that knowledge of physics and thermodynamics could be used to rule out something like a 4-second mile.

I think mirror neurons are a good example. I still don't understand what difference it makes which neurons are involved in empathy. Pretty much every signal in the brain can be tuned out or strengthened, right?

In what sense does the right hemisphere "see the whole without breaking things into pieces"? Especially when it's implemented on a distributed computing medium like neurons.

Your last paragraph seems to make a more concrete claim, which is that we can tell which kinds of cognition are more or less draining, analogous to native vs emulated execution. This makes sense to me, but it's not clear that we couldn't already tell roughly how draining different modes of thought are without neuroscience, nor that the main claims of Eastern spirituality depended on how draining these different states are to hold.

In any case, thanks for clarifying.