Effective Altruism’s obsession with AI safety helps bury bad behavior
bloomberg.com
bloomberg.com
Do we believe that we alone, of all people who have believed in the coming end, have it right this time?
And since the world exists every group will be wrong except one.
I think the risk of human extermination is much lower than many in this group believe because I think AI is much more compute bound than algorithm bound. So I think a hard takeoff is very unlikely. But while the median case for climate change is probably worse than the median case for AI. The existential risks for AI seem much higher. And it's a concern worth taking seriously.
They don't. They hardly talk about anything else except AI. From the outside it looks a lot like an obsession.
Personally I think they treat AI as an existential threat above all others because they consider themselves to be the pinnacle of intelligence in the universe and AI is showing that human intelligence might not be so special after all. So it's an existential threat to their sense of self. Other existential threats merely threaten their lives and are therefore not as important.
This would be fine, if acknowledged with a degree of self awareness. But it's not, instead it's wrapped up in so many layers of convoluted logic that most people, insiders and outsiders both, can't see the real reason for this obsession.
Just looking at the topics section of the EA forums (https://forum.effectivealtruism.org/topics/all), we've got
- 466 posts in AI safety
- 1060 posts in existential risk (seems like a roughly even mix of pandemics, nuclear war, and AI)
- 816 posts in global health
- 193 posts in animal welfare
- 42 posts in biosecurity & pandemics
It would be interesting to understand how much of the "existential risk" category is devoted to biosecurity or pandemics, and how recent those posts are.
Pending the above, my tentative takeaway from this is that COVID - arguably the greatest threat to human health and wellbeing in the past several decades - seems to have been a minor/non-existent blip on their radar until it actually happened. This raises serious questions in my mind about the community's predictive abilities and/or bias.
It looks like the Forum is confusing here: it has "Biosecurity & Pandemic Preparedness" [1] as one topic, and "Biosecurity" [2] as its largest subtopic. But when you click on the parent topic it's not showing the subtopic posts; I've filed a bug.
Older good biosecurity posts include:
* 2014: Good policy ideas that won’t happen (yet): https://forum.effectivealtruism.org/posts/n5CNeo9jxDsCit9dj/...
* 2017: Biosecurity as an EA Cause Area: https://forum.effectivealtruism.org/posts/eXgCREwL5PQWHEtvm/...
* 2021: Concrete Biosecurity Projects (some of which could be big): https://forum.effectivealtruism.org/posts/u5JesqQ3jdLENXBtB/...
[1] https://forum.effectivealtruism.org/topics/biosecurity-and-p...
"Some of the reasons I'm skeptical of natural risks are that first of all, they've never really happened before. Humans have obviously never been caused to go extinct by a natural risk, otherwise we would not be here talking. It doesn't seem like human civilization has come close to the brink of collapse because of a natural risk, especially in the recent past.
"You can argue about some things like the Black Death, which certainly caused very severe effects on civilization in certain areas in the past. But this implies a fairly low base rate. We should think in any given decade, there's a relatively low chance of some disease just emerging that could have such a devastating impact. Similarly, it seems like it rarely happens with nonhuman animals that a pathogen emerges that causes them to go extinct. I know there's one confirmed case in mammals. I don't know of any others. This scarcity of cases also implies that this isn't something that happens very frequently, so in any given decade, we should probably start with a prior that there's a low probability of a catastrophically bad natural pathogen occurring."
I wonder if this is a case of extrapolation fallacy? Modern human dynamical behaviour is significantly different to both animal behaviour and less modern behaviour. Viruses spread more easily in the age of global travel; those that used to be too deadly to spread very far suddenly have more opportunity.
EDIT: Reading this more carefully, the speaker does actually address globalisation, but seems to dismiss it as a counterargument and I'm not really sure why.
This is a very low bar to jump over!
Anyway I read through that article. There's a lot of guff about Newcomb's paradox and Eliezer, and a single paragraph referencing COVID, with two links supporting your statement. I've clicked through those links and have come out unimpressed.
The first link is used to supported the claim that "The rationalist community was well ahead of the public": https://slatestarcodex.com/2020/04/14/a-failure-but-not-of-p... .
Scott rightly highlights how the media and institutions got things wrong, but conversely gives only a few examples of "generic smart people on Twitter" getting some things right in the early days of the outbreak. He confesses that he himself did not predict the seriousness of COVID.
The second link is used to support the claim "the wider tech community, were using masks and stocking up on essential goods, even as others were saying to worry about the flu instead": https://putanumonit.com/2020/02/27/seeing-the-smoke/ .
The linked article was written at the end of February, when panic-buying had already firmly set in across the US.
There is nothing here about the reasoning or predictive abilities of the rationalist or EA community specifically. Nor is there any compelling comparison of its response compared to the wider public.
I know I'm jumping the gun just a little, but c'mon. An apocalypse list without climate?
The topics listing isn't great, but there's been a lot of discussion about climate change on the forum: https://forum.effectivealtruism.org/topics/climate-change (239 posts)
The AIpocalypse seems incredibly unlikely. I'm a lot more worried about the nukes, and even if we end up with robotic overlords, I'd bet they'll be a whole lot better at administration than meat -admins have proven.
I'm with you on engineered superviruses. Feasible, likely, and incredibly high impact.
What I kind of keep coming back to is the risk profile of all this stuff. That magic vector of likelihood * impact. Global warming is happening now. And it's real bad - worse than I think we give it credit for.
What I worry about is that we're the proverbial frog in the pot. Things get just slightly hotter each year, so we'll miss it when we actually boil.
1 https://www.google.com/search?q=double+asteroid+redirection+...
My comment was written with the summary of that article in mind (I didn't make this clear):
> Climate change is going to significantly and negatively impact the world. Its impacts on the poorest people in our society and our planet’s biodiversity are cause for particular concern. Looking at the worst possible scenarios, it could be an important factor that increases existential threats from other sources, like great power conflicts, nuclear war, or pandemics. But because the worst potential consequences seem to run through those other sources, and these other risks seem larger and more neglected, we think most readers can have a greater impact in expectation working directly on one of these other risks.
There is an excellent chapter on the existential risks associated with climate change in Toby Ord's book The Precipice, which you can get a free copy of at https://80000hours.org/the-precipice/
The way the EA forum handles parent topics is confusing, but this is only counting posts tagged with the top-level label. It has several subcategories, and there are hundreds of posts when you include them:
* 218: Biosecurity
* 176: Covid
* 54: Pandemic preparedness
* 40: Global catastrophic biological risk
* 39: Vaccines
* 20: Biotech
* 16: Life sciences
* 11: Dual-use
* 10: Biosurveillance
(Posts can have multiple labels, so the above list double-counts a bit. I don't see an easy way to extract the actual count from the forum, but it's at least 218, the count for the largest category.)
How many EA people do you know?
That seems to get the most clicks, but I usually associate them with things like mosquito bed nets to help prevent malaria, because that's a far more common topic in my experience.
I'm sure you can find particular people who are all about the AI risk stuff that seems pretty wild to me, too.
But when you say "focused exclusively" I feel like you may have mostly seen news articles about wild AI stuff and not the average EA enthusiast who spends their time talking about picking efficient charities.
That isn't to say that the malaria stuff isn't still a big topic, but there has been a shift.
Longtermist topics are the perfect combination of high-prestige and smart sounding with a complete lack of accountability, because the impact of research or projects won't materialize for decades. Ironically, in practice it directly contradicts the founding tenants of EA, despite being a logical conclusion of the underlying moral philosophy.
But since the real goals of the EAs are not to do any of that (it's just a bunch of socially-awkward geeks looking to feel accepted in some kind of community and find sexual partners; in other words, trying to feel like normal humans), they fooled themselves into believing that such short term actions are "ineffective"; actually, any actual measurable hard work has to necessarily be ineffective, or else it would actually have to be done. So they keep pushing their agenda to longer and longer term issues, more abstract and less well-defined as possible, which cannot be actually measured, so they can continue to build this weird, meaningless, ridiculous fantasy that they live in (and which supports their social hierarchy) without having to actually do anything concrete.
That's really not true. There's a huge overlap in these groups, but the top EA charities are not shoveling all their money at AI alignment orgs... even if one can argue that they should.
https://www.effectivealtruism.org/articles/introduction-to-e...
https://www.lesswrong.com/posts/JmmA2Mf5GrY9D6nQD/four-focus...
https://www.givingwhatwecan.org/best-charities-to-donate-to-...
Even if that were true, there are lots of people already looking at those other existential risks. The EA crowd was the only group concerned about AI for a long, long time. Seeing an existential risk that nobody else does leads to evangelism and hyper focus so people start paying attention.
Any reason to believe this? The human brain runs on 20 watts (!). Obviously biological hardware is not directly comparable to silicon, perhaps silicon is intrinsically less efficient by some factor such that in order to obtain 20W of human brain compute we need to expend K times more in silicon. But is that factor really so large that we are compute-bound? How big would K have to be for us to be compute-bound anyway? If you could have Einstein for 200kW (i.e. silicon is intrinsically 1000x less efficient than brains) would that be compute-bound? It seems much more likely that we are algorithm-bound and simply not using our available compute efficiently.
Yudkowsky's hard take off mentions "solving protein folding", nanomachines, persuading everyone to let them connect to the internet, infinite adaptability, etc. I think those are unrealistic, but obviously they still have a non-zero chance.
More probably bad possibilities are AI-Hitler (monomaniacal populist absolute leader) or AI-Stalin (manipulative, smart, absolutely paranoid, rising through the ranks) ... so something that's human-like enough to be able to connect with humans, to manipulate them, but at the same time less affected by the psychological shortcomings. (Ie. such an AI could spend enough time to cross-interrogate every underling, constantly watch them, etc.)
And yes, a very efficient immortal dictator is very bad news, but still bound by human-like limits.
And the big infinite dollar question is could this hypothetical AI improve on itself by transcending human limits? Let's say by directly writing programs that it has conscious control over? Can it truly "watch" a 1000 video streams in real-time?
Can it increase the number of its input-and-output channels while maintaining its human-like efficiency?
Because it's very different to run a fast neural network that spits out a myriad labels for every frame of a video stream (YOLO does this already, but it's not 20W!) and to integrate those labels into actions based on a constantly evolving strategy.
Sure maybe the hypothetical AI will simply run a lot of AlphaZero-like hybrid tree-search estimator-evaluator things ...
Anyway, what I'm trying to say is that our 20W efficiency comes with getting tried very fast, and using "fast mode" thinking for everything. (Slow mode is the exception, and using it is so rare that we basically pop open a glass of champagne every time.)
>And the big infinite dollar question is could this hypothetical AI improve on itself by transcending human limits? Let's say by directly writing programs that it has conscious control over? Can it truly "watch" a 1000 video streams in real-time?
Even if its mind wasn't truly directly scalable, it could make 1000 short or long-lived copies of itself to delegate those tasks to.
AI is...well, people losing their shit over ChatGPT (**) aside, AI is not going to be real enough to worry about for a few more decades at least.
(*) Anyone who's about to regurgitate some fossil-fuel industry talking points in response, just save your breath.
(**) Above note goes double for this.
No, I would rather we wait until we are close enough to AI that we are talking about something concrete, rather than making wild speculations.
Would you have opposed research into renewable energy in the 1970s since global warming was still a few decades away from being something we needed to worry about?
If you think this than you have misunderstood me. Now go elsewhere and bother other people.
Except at most one.
What’s the upper bound for climate change’s existential risks? The end of human existence and society as we know it except for a relatively small number of survivors living in a world those of us alive today can barely imagine?
https://www.scientificamerican.com/article/impact-from-the-d...
When the article was written, CO2 concentration was 385 ppm. Today it's 421 ppm. If CO2 concentrations were to rise linearly, the timeline for reaching 1,000 ppm would be 250 years. Achieving that would require emissions to stabilize at ~2014 levels. If emissions keep rising, then perhaps it's closer to 150 years. If strong feedbacks kick in, like methane from melting permafrost, then maybe 100 years or less, or maybe we'll reach that happy 2500 ppm mark and see what a real mass-extinction looks like.
Of course the ocean has a big thermal mass, so it will probably take quite some time to heat up enough to trigger the disaster even after we reach that level of atmospheric carbon. Hopefully everything will be fine?
It matters a lot. Whilst I was eating dinner I've now read the article you cited, the paper that it cites (Kump, Arthur & Pavlov, 2015). I also cross checked the argument and numbers against several other papers. Ward's argument is extremely slippery. This is sadly what I'm coming to expect from academics with unverifiable models. You have to read everything they write adversarially.
"In [Kump2015]'s models, if the deepwater H2S concentrations were to increase beyond a critical threshold during such an interval of oceanic anoxia, then the chemocline separating the H2S-rich deepwater from oxygenated surface water could have floated up to the top abruptly. The horrific result would be great bubbles of toxic H2S gas erupting into the atmosphere."
Observe that his argument starts with a "critical threshold" for H2S levels, not CO2, and that he doesn't tell us what this critical level is. The obvious questions are thus: what is this level, is it realistic for our present day oceans to reach this critical level and if so, how? To get to those levels of H2S he hypothesises that:
"if ancient volcanism raised CO2 and lowered the amount of oxygen in the atmosphere, and global warming made it more difficult for the remaining oxygen to penetrate the oceans, conditions would have become amenable for the deep-sea anaerobic bacteria to generate massive upwellings of H2S."
In other words in this theory the supervolcanoes have to come first, triggering a sharp drop in oxygen levels in the atmosphere, which in turn then causes the chemocline to move, which then causes the H2S upswellings. It's all caused by a massive loss of oxygen, not increase in CO2 levels, which is simply another result of the volcanisms. Kump2015 also makes it clear that their hypothesis requires a truly massive drop in oxygen levels to occur. We'll look at how much in a moment.
But just a few paragraphs later Ward has forgotten all about the volcanoes and oxygen levels. Suddenly it's all about absolute CO2 levels - not even rates! Cause, effect and unrelated side effects have become entirely muddled, probably because he knows nobody would care about his article unless he ties it to global warming armageddon somehow and because hey, this entire field is nothing but assumptions, suppositions, and playing with numbers that can never be verified anyway so why not?
His argument is brittle in other ways. He asserts without any backing argument that whilst supervolcanoes explain all the other non-asteroid mass extinction events it doesn't for the Permian because, apparently, supervolcanoes are really great for plants on land which can "probably" survive the warming. And there was me thinking that large scale volcanic activity is supposed to be very bad for plants because it dims the atmosphere:
https://www.vulkane.net/en/volcanism/supervolcano.html#:~:te....
"Plant growth is restricted and mass extinction can be caused."
Back to the H2S claims. Turning our attention to the Kump2015 paper we find immediately that it's got an annoying structure in which they work backwards from their desired scenario to calculate the level of H2S that could trigger it:
"Thus, if the H2S of the deep sea increased during an anoxic interval beyond a critical value (1 mmol/kg), upwelling regions of the world ocean would become sulfidic, even with the modern [oxygen level at atmospheric pressure] .... A slightly more sulfidic ocean with H2S = 3 mmol/kg ... could sustain ... 2000 times the present-day flux and a critical value for the atmosphere (see following)."
But these critical values are never placed clearly in context. Is 3 millimols a lot or not much? Note that the 3 mmol/kg value is the absolute "best" case for their scenario; it can only be that low in (they estimate) 0.1% of the world's oceans because normal ocean requires levels 20x higher. They do admit that:
"The [H2S] condition is extreme, and thus likely to have been rarely achieved in Earth history. Is there any evidence that such conditions have occurred in the geologic past?"
How extreme is it? The value of the normal ocean would be important to have here but they don't give it to us. The paper "Hydrogen Sulfide in the Black Sea" by Volkov & Neretin does give values though, and bear in mind this is by far the most anoxic basin in the world: at a depth of 1km the value is ~314 micromols/kg. So these H2S levels that are claimed to trigger this process are literally many, many orders of magnitude higher than even the Black Sea.
The levels of oxygen drop needed is something they also don't seem to directly share, but in one section they're kicking around a figure like half of all today's oxygen having vanished, or for a different event, 99% of all oxygen having gone.
Even putting aside that these papers are just piles of completely unverifiable suppositions stacked like a jenga tower, there is absolutely nothing even remotely close to realistic about this scenario happening to us. It requires oxygen levels to drop so much that if it were to ever become an actual threat we'd all have died of oxygen starvation long before.
Still, I certainly do hope that you're correct! And thank you for the thoughtful and well-considered reply!
Since very smart human beings have in history done a lot of damage, but never ended the species or anything, we can conclude that the world has an "intelligence safety margin" that extends up to Alexander the Great or Napoleon. There have been some incredibly smart people in human history and none of them have ruined everything, a few countries at most.
At best, history serves as a counterexample to the idea that if an AI goes bad the attendants would just unplug it, seeing how often it is that dictators don't get stabbed by their aides as soon as they start causing mass deaths, instead often receiving broad popular support as the world burns.
This has been discussed under "warning shots" https://www.lesswrong.com/posts/idipkijjz5PoxAwju/warning-sh...
Gain of function research is supposed to help with pandemics, right? Did it help with COVID?
No, not only has GoF not helped with COVID and probably caused it, but that's also true of the entire field of virology.
Every step of this relies on assumptions that are not merely questionable, but unfalsifiable.
Whether AGI is possible or not, regardless of anyone's personal opinion, is as yet unprovable and unfalsifiable.
Assuming that AGI itself is possible, there is no way to tell whether we, as humans, can create an intelligence that is "smarter" than we are.
Assuming that we can create an AGI that is "smarter" than we are, there is no way to determine whether it would be able to upgrade itself to become exponentially smarter than that, and beyond human understanding and control.
If you have a hard time coming up with arguments against these things, maybe it's because they're fundamentally unfalsifiable, which makes them useless for trying to build any framework of understanding on.
But the AI safety groups don't just assume these without any justification. The existence of the human brain itself is either a proof and very strong launching point for most of these. In 1930, maybe you could have convinced someone that a nuclear bomb is impossible or at least an unfalsifiable worry, because one had never yet been built, but your reasoning is like trying to cast doubt on the possibility of artificial digestion, when there are already billions of stomachs roaming the earth.
The idea that a computer can't possibly be made to accomplish whatever a brain can is losing plausibility with every passing day. And as for surpassing it: a human brain is powerful, but has so many surmountable limitations, like: requiring 20 years of education to get up to speed with existing experts; dying after 70-90 years; not being able to run copies of oneself in parallel; etc. We only need to imagine removing these limitations, and doing so violates no known scientific principles.
It's like having a 50-megaton warhead in your lab, and meanwhile the rocket scientists are gradually figuring out how to make rockets fly, and you're saying "yes this is a big bomb, but the idea that one of these could be more dangerous if mounted on a missile is an unfalsifiable assumption!"
Consciousness, intelligence, sapience—these are not well-understood phenomena. We don't know what makes us conscious. We don't know if other animals are, or to what degree. It's not even possible to determine with any scientific certainty that another human being is conscious.
As things stand, "we can build an AGI" is not a scientific statement. It is not grounded on a foundation that allows clear reasoning about it, one way or the other.
Your arguments are not incorrect; however, they do not bridge the gap to "and so we can definitely make a conscious, sapient, intelligent computer." Being able to replicate particular capabilities of the brain is not the same thing.
And, again, that's only the first unfalsifiable proposition that must be satisfied in order for the purported AI threat to be real. They also have to be capable of breaking free of our control, decide we're a threat to them for whatever reason, and have the means to carry it out.
Consciousness is irrelevant, as it's clearly not necessary for AGI nor even human intelligence; otherwise you wouldn't say "it's not even possible to determine with any scientific certainty that another human being is conscious."
On what grounds do you believe that there are capabilities of the brain that cannot be replicated by a computer? When I look at the 1.5 kgs of matter in a typical human brain, I don't see anything that jumps out and says "my operation is not computable!"
At least with fusion, the high temperatures and pressures are a clear barrier. Our brains don't need to be held at 3.8 trillion psi and 15 million Kelvin in order to enjoy poetry.
I can certainly argue on the other points as well (breaking free / deciding threat / acquiring means) but you need to pick your goalposts one at a time. I think the first step would be asking yourself: if you were a super-genius but being held by guards in solitary confinement on a remote island with only a supercomputer connected to the internet, how would you earn money?
Those conditions clearly would not have stopped even an ordinary Satoshi Nakamoto from gaining control of sufficient resources to hire private military contractors to arrange his escape. I'm not sure what a superhuman would do, but that's a human baseline.
The argument goes like this: If you want to save the world, then doomsday scenario S is the best place for you to invest your resources if S has a nonzero probability, and currently we are investing less resources in S than in other existential risk scenarios (per basis point of probability).
"AI risk" is a pretty good candidate for S (especially back in 2010-2015 when the movement was just starting.)
> Whether AGI is possible or not, regardless of anyone's personal opinion, is as yet unprovable and unfalsifiable.
"AGI is impossible" is certainly falsifiable: All I have to do is build an AGI and show it to you.
Further, there is no theoretical reason AGI is impossible. Rather the reverse; consider these Well Settled Scientific Facts:
- (1) Human Mind = Human Brain
- (2) Human Brain obeys the laws of physics
- (3) We understand the laws of physics well enough to simulate them in a computer
If you accept these three facts, then in theory AGI is possible: You could implement AGI by building a machine implementation of the Human Brain by fully simulating the underlying physics.
> Assuming that AGI itself is possible, there is no way to tell whether we, as humans, can create an intelligence that is "smarter" than we are.
> Assuming that we can create an AGI that is "smarter" than we are, there is no way to determine whether it would be able to upgrade itself to become exponentially smarter than that, and beyond human understanding and control.
An AI safety person would say to this: "You're right, we don't know -- and that's exactly the problem."
What we can do is assign a probability based on our confidence. How likely do you think it is that we can do those things? 20%? 2%? 0.000002%?
If you say "0.000002%", what makes it so extraordinarily certain it's impossible? If you say "2%" or "20%" then as a matter of self-preservation, shouldn't our society be devoting a lot of money and smart people's time and attention to figuring out how to make sure it doesn't happen?
As well, while the arguments are logical, most of them rely upon large assumptions to move between steps. If any of these assumptions fail, the entire thing fails. Especially the hard take off assumption.
As well, the assumption that AGI is happening in the next 3-10 years. I’d say most prominent people in the AI research space don’t think we’re much, if any, closer to AGI. Yet you have Yud and LW screaming that we will all be dead in a few years and AGI is right around the corner.
When people like Chollet and Ng say we aren’t close to AGI, I’m more likely to believe they’re right, vs. Yud who hasn’t contributed to any actual developments within the field besides theorizing about alignment and how AGI can go wrong.
So unless one is proposing a global crackdown on AI research, AI safety is a lost cause.
There are lots of solvable problems that will never be solved if we just throw up our hands and say "we'll never get it right, might as well not try".
Any uncontrolled group that creates an AI can either find a body of safety research accessible to them, or not. Preparing the former is hardly a lost cause.
AI certainly has risks, I don't think any reasonable human being doubts that, its just that the AI doom cult seems to think the worst outcomes are near certainties without really backing that up.
What are you basing this on? There have been tons of arguments written about why these worst outcomes are likely. Read Bostrom's Superintelligence for example, or Yudkowsky's Intelligence Explosion Microeconomics.
Many conversations with AI doomers. They gloss over and make assumptions about intelligence that aren't really backed by priors and when this is pointed out they hand wave and say "but computer".
> Read Bostrom's Superintelligence for example, or Yudkowsky's Intelligence Explosion Microeconomics.
I don't really have any interest in doing so, and if I'm honest have a particularly unfavorable read of Yudkowsky as a person based on his cultish following.
The orthoganility thesis is an unproven assumption that intelligence and goals are not corellated, meaning an intelligent being can pursue stupid goals. States like that, it’s obviously wrong and laughable. But by using complex language, EA cultists hide the ridiculous assumptions their system has so that they can maintain their feelings of superiority while gaining real power that enables them to abuse others.
You could conceive of a super intelligent AI that came into existence with the goal of terminating itself. That would be an "stupid goal" from our perspective, since we have the goal of self-preservation really ingrained in our brains.
But for a being that self-termination is the absolute best thing ever, it's not stupid. It makes perfect sense, since, well, that it's goal. It doesn't care about self-preservation, it doesn't care about becoming more intelligent/rich/powerful, other than as an instrumental goal to help achieve self-termination, if it's not able to do so in its current state.
And most importantly, no amount of getting more intelligent would change this fundamental goal, just as humans getting more intelligent has not overridden our fundamental goals of "breathe, feed, have sex". It may have given us other goals as well, but those are very much still there.
Yes, that's pretty much the point.
And most likely, humans will have a pretty naive understanding of whatever motivates a superintelligent AI.
This is just a specific example of why I reject the orthogonality thesis. You change the context, you educate the agent, you change the goals of the agent. I do not agree that humans only chase “breathe feed sex” and while I do believe many stupid behaviors do come from evolutionary history, It’s plainly obvious that education, training, and genes play a role in self restraint and goal redirection.
And I agree that humans do not only chase "breath feed sex", as I explicitly said that in my comment. We have other goals as well. But those are very much still there.
"Better" according to what metric?
It may be the case that there is a tendency for high-intelligence humans to pick "more enlightened" goals. Perhaps there is a natural "enlightened goals" attractor for our species.
However I don't think we can extrapolate from that to a fundamentally alien AI.
I think even if this statistical tendency exists, it has clear counterexamples -- consider that 2 genius chess players may have opposite goals, of beating one another. And we shouldn't bet the future of humanity on this statistical tendency extrapolating outside of the original distribution of human species.
Here are some intuition pumps on how diverse goals can be even across intelligent species:
* Orcas killing sharks for their livers: https://www.livescience.com/2-orcas-slaughter-19-sharks-in-a... I don't believe dolphins show the same level of violence, even though both are smart cetaceans
* Intelligent dogs bred to be responsive and attentive to human needs -- unlike close cousins like the wolf
* Chimpanzees and bonobos are both related to humans, both highly intelligent, but with very different culture and goals https://www.youtube.com/watch?v=c6Ko0Hzi47U
As soon as you phrase goal selection in terms of metrics, you’re assuming that goal selection is based on some other goal - that is you’re already assuming the orthoganality thesis. Your logic is fully circular.
One thing that’s interesting to note about all of the examples you picked - every single one of those species shows cooperative behaviors. They share many other behaviors that are more similar than they are different. To reject the orthoganality thesis it’s sufficient to show that there is an empirical general association between intelligence and certain goals - then we can extrapolate an AI although of course it will function differently and may have many unusual behaviors will tend to follow those goals more directly. For instance intelligence is associated with : cooperation, empathy, inter and intra species communication, curiosity, etc. all of the species you mentioned exhibit these more than less intelligent species. Meanwhile something like “hunting to eat” is observed across the intelligence spectrum.
I maybe agree with you that there's an this belief that a maximally intelligent creature will blindly follow maximally obviously stupid goals, and that belief is under-argued, but your phrasing above isn't the slam dunk argument that you seem to believe.
To me it's obviously wrong when stated that way because 'stupid' is not an appropriate metric for goals. We may consider goals good or bad within our value system, but that has little do to with 'stupid' or 'intelligent.' e.g. a body builder may have a goal to get as ripped as possible, a VC to make as much money as possible, an ascetic to deny the flesh as much as possible. Which of these are stupid or intelligent goals? I don't think that's a question that makes sense.
We may consider the goals of the Athenians (to expand their power) "worse" than the goals of the Melians (to be left alone)[0], but I don't see how they were "stupider."
[0]: https://en.wikipedia.org/wiki/Siege_of_Melos#The_Melian_Dial...
A safety mindset would suggest that rather than disregarding it until proven true, we should worry about it until proven false.
> meaning an intelligent being can pursue stupid goals.
It would pursue very intelligent instrumental goals, but the terminal goal is a free variable and I don't think there exists any measure by which terminal goals can be considered smart or stupid. It would be whatever is implied by its programming.
> States like that, it’s obviously wrong and laughable.
Perhaps not so obviously wrong nor so laughable as you think?
There’s no clear delineation in real entities between instrumental and terminal goals.
Yeah if you’re talking to a brainwashed religious fanatic, definitely it can seem counterintuitive to the doctrines they believe.
In healthy individuals, yes there absolutely is. Terminal goals are the ones you pursue for their own sake; instrumental goals are the ones you pursue as part of a plan to pursue a terminal goal, or another instrumental goal which connects to a terminal goal. Most people go to work in the morning not as a terminal goal, but as an instrumental goal; employment is in service of another instrumental goal of earning money; earning money is in service of a terminal goal of not starving to death. This is not exactly controversial stuff here. Some people do get so focused on an instrumental goal like "earning money" that they develop tunnel-vision and forget what terminal goal that money was originally in service of, but that's something most of them will eventually realize and then write a self-help book about.
Anyway, it takes intelligence to decide what your instrumental goals should be, such as whether there's perhaps a cleverer way to make money than by going to work for your boss each morning, but there's no way in which intelligence will help you choose your terminal goals. For the most part they aren't something you can even consciously choose.
In reality, people do not need to go to work to “not starve to death” as you say. There are a myriad of ways to survive without working a daily job.
Humans have to be socialized and trained to work a 9 to 5 job - there’s an entire education system structured to help create humans who view that as an acceptable goal.
No what you are saying may not be controversial in your little community but the AI panic is mostly isolated to a small community in a small corner of the USA.
As for there being "a myriad of ways to survive without working a daily job", congratulations! Your intelligence has allowed you to identify alternative instrumental goals that provide a path to your terminal goal; now you can rank them and choose the best option. You can also grow carrots in the garden or ask your neighbour if they have any, or ask your spouse to pick some up on the way home. Your intelligence will do the work and find a way. But your intelligence isn't what will guide you toward preferring carrot soup over parsnip soup, and preferring parsnip soup over fasting.
I'm not American and not in the USA, btw.
Think more carefully about the implications of multiple ways to survive here. Why do people pick one over the other? In a terminal/instrumental goal model, agents would pick the instrumental route that maximizes the return on the terminal goal. In reality we see that instead humans adopt habits, processes, and heuristics that guide them through daily life even when those do not lead to any specific goal.
So remind me of your original point? I believe you said it's "obviously wrong and laughable" that "an intelligent being can pursue stupid goals". Now here you are trying to convince me that humans are the ones who, like Pavlov's dogs, "pursue habits, processes, and heuristics that guide them through daily life even when those do not lead to any specific goal". Even when those habits involve repeatedly re-opening a fridge that you already know has no carrots in it, or salivating at a bell when you already know no food is coming.
So I'm confused how that proves your point about AGI. If I accept your view, it seems that if an AGI does merely no better than a human on this metric, I should anticipate all sorts of strange and irrational behaviour, including the pursuit of goals that would appear stupid, such as addiction to a reward channel. That does not seem to undermine the orthogonality thesis.
And the smarter the AGI gets, presumable the less it should lean on Pavlovian heuristics and the more it should make use of clear thought, which puts it more in my camp.
So that would apparently put the lower bound at "the AGI takes unexpected and irrational actions because it's not a rational agent and doesn't think coherently", and the upper bound at "the AGI takes unexpected and dangerous actions as rational steps toward an unaligned terminal goal".
I'm not sure where in this chain of thought it becomes laughably obvious that intelligence and goals are correlated, such that an AGI's increasing intelligence will tend it toward actions that we humans approve of, because anything else would be a "stupid goal"?
Let chaos take the world! /s
So yeah an apocalypse cult could be right, but it would necessarily be for the wrong reasons. Just like a broken clock that matches the current time 2 minutes of the full day.
*for what it's worth I do not find it convincing but I am arguing "even if I found it convincing, I wouldn't believe it"
I've had to update accordingly, and now I'm less skeptical that the barriers ahead will be harder to break than the ones behind us.
because we are still here
This is not an argument, it is a tautology powered by the anthropic principle.
but yes, there most certainly were people predicting it and they probably didn't get a chance to enjoy being right.
We’re creating a fundamentally alien intelligence from scratch and basically just crossing our fingers and hoping to hell it cares about keeping us alive.
Just because the parts of the community in the Effective Altruism / AI Safety have developed aspects of a cult doesn't mean that their arguments are wrong. It just means that they're humans doing human things. It sounds like their main mistake was to not realize that the cult-like behavior of other cults was sociological, due to the fact that they contained humans, as opposed to from their belief systems; and thus didn't take any steps to try to prevent their own movement from developing cult-like aspects.
That said, if you have a movement of thousands (?) of people all trying to figure out how to make AI safe for over a decade, and you still don't feel like you're any closer to that goal at the end of it, then you should probably step back and ask yourself what you're doing wrong.
If you're one of the folks doing this at least acknowledge the story would have been quite boring if it was 800 pages of "everything went well, the AI turned out really great, very helpful. Would use again"
If you're not blindly afraid of zombies you shouldn't be blindly afraid of AI. Give it thought.
If you're still scared after dismissing sci-fi bias, cool, I'm all ears.
I'm looking into MIRI, if that's not "the AI safety community" please do correct me.
I've become very aware ever since ChatGPT started saying inaccurate things confidently that humans have a much worse hit rate.
Seriously, keep an eye out for it, you'll see it everywhere. At least ChatGPT will double check if you ask if it's sure. People tend to just get annoyed when you don't blindly trust their "research" haha.
While MIRI is prominently connected to Yudkowsky, I wouldn't treat them as defining the AI alignment community. There are many people not involved with it who make substantive posts and discussions on LessWrong and the Alignment Foundations forum. There are other organizations too. OpenAI considers alignment important and has researchers concerned with it, though Yudkowsky argues the company doesn't do enough to prioritize it relative to the AI progress they make. Anthropic is an AI company prioritizing AI safety through interpretability research.
I remember feeling like this (brain CPUs pegged at 100%) trying to slog through HPMOR the first time too in fairness, it's just too many concepts to take in in one sitting. I'll get there eventually if I keep at it but not on my first read.
I'll consider my opinions on AI safety void due to lack of knowledge for now, always try to jump over the first stage of competence. I'll start with the Wikipedia page for AI alignment, haha.
Thank you for your responses in any case, I'll dig into this further!
I need to know a lot more on this subject before dismissing it! I'll give that a read too, thank you :)
Waiting for the day our Overlords make Graphic Card 2.0 and refuse to sell any to the plebs 'cause the overlords "know better"
My biggest problem with AI safety is that, simply, the problem they envisage doesn't exist yet (generally, at a minimum, relying on the existence of "AGI"). Hence discussions about it have to make a huge amount of assumptions about a whole range of aspects of what the AI threat will be - what the AI will be capable of, what it's impact will be - before getting on to what possible solutions might be relevant to preventing it. But given the first two are so undefined, the later is pure speculation - one that is difficult to criticise directly, because any specific critisms can usually be easy deflected by adjusting any of the above assumptions without making a substantial change to the "inevitable" conclusion.
That's why it feels like an apocalypse cult to me - it's a conclusion, that has little strong evidence today, stacked on top of a constantly shifting set of assumptions, allowing adherents to avoid backing their arguments with evidence.
As long as it cannot also explain its own decisions transparently [1], then there there will always be a need for AI safety.
One of the likely outcomes like that is even stronger pivoting towards walled gardens and end to end stuff than we previously had on the Internet. Can't train a model on someone else's encrypted database without decrypting it first.
Addendum: nor his body.
Several of the accounts turned out to concern the same person, Michael Vassar, who was central ~10y ago but kicked out for this and other harmful behavior, and now describes himself as anti-EA.
On the other hand, the comment treats the cases they can't identify as not concerning from a community perspective ("it's very unclear from this what actions anyone could or should have taken") and I don't endorse that.
All that said, the amount of energy we expend on AI safety probably shouldn’t be zero. But I choose to believe that if Roko’s basilisk is going to torture us over anything, it’ll be that we didn’t raise up millions of AI scientists from the cotton fields and sweatshops. Credit to the less prominent altruists doing that, or who at least have some form of compound interest built into their utility function which compels them to be kind, now.
Since the A.I. would have to already exist (and there's no reason to believe it would be incentivised to even encourage humans to create other, rival A.Is, with potentially differing goals) before it could punish anyone - Q.E.D.: Roko's basilisk has essentially zero likelihood.
Worse - it's an essentially christian eschatological idea. What is this hypothetical A.I. but an avenging angel, a righteous judge at the end of time? It's pure religion. Dark, manipulative, blood sacrifice religion at that.
A similar argument regarding Roko's Basilisk is that it is effectively a prisoner's dilemma, where the Basilisk torturing us and us not helping it is the Defect-Defect scenario. It makes no sense for either party to defect.
> except in the context of altering their future behaviour.
It really seems like a very contrived version of Pascal's mugging.
AI safety isn't about Roko's basilisk, or the idea that AI will take revenge on us for our moral choices. It's about the risk that an AI smarter than us will be made and built to pursue some goal without caring about our well-being.
We already have those and they're called corporations. They've done significant real damage to the world already and they are still working hard to do more damage.
It makes little sense to me to focus on this potential future problem when we haven't even agreed to deal with the ones that we already have.
It is not for lack of trying, corporations modify themselves all the time. That they are quite often unsuccessful at making meaningful improvement should be taken as a prior against the idea that an AI will inherently be better at it.
It's also about the risk that AI is dumber than what we expect it to be and fails to properly solve the issue.
A sufficiently smart AI should be able to understand our intentions and - assuming it is aligned with us - attempt to fulfill that purpose.
Dumb "AI" will fail in ways that don't actually solve the task, or solve it only on the surface / do reward hacking.
If we don't figure out alignment, we can't make this assumption. This is the part that AI safety people are concerned with making reality.
An aligned agent will not fall into stupid errors, thus solving the lower bound of performance is a necessity for complete alignment.
A task such as "bring me a cup of tea" has lots of implied information such as not making it too hot to burn the person, not breaking anything in the process, not harming other people in the process and so on.
1. https://aiascendant.substack.com/p/extropias-children-chapte...
I’m pro-American because I believe in its stated values. Moreover, I believe that despite original and continuing failures to perfectly reflect those values, there is large and concrete progress in aggregate.
I feel the same way about EA: I aspire to their ideals (including the continued discussion and reconsideration of those ideals) and believe the weight of thousands of people quietly moving concretely in the right direction far exceeds the damage of bad actors (who would have twisted any ideology they were immersed in).
So the bad actors should be punished. The parts of the community system which failed should be changed. AND the many others who are earnestly trying to live according to these values should be judged according to their _own_ impact.
I disagree and think both the US and institutions like EA are too tainted. The US hasn’t done anything about slavery except make things worse still. That’s not on the founding fathers any more only. It’s on the US since the civil war. EA will take money from future multimillionaire and billionaires. Much like the US, they are both completely controlled by capitalism. I am in the US but not pro US.
> The full post goes into more depth and nuance, but it's worth stressing how heavily Leverage relied on “debugging,” based on Anders's “Connection Theory.” Debugging consisted of — to paraphrase and oversimplify — opening up one's psyche, history, self, and emotional core to maximum vulnerability, generally to one's hierarchical superior, and then doing all you could to follow their suggestions to “fix” your mind.
Any hoo - this stuff is the tip of the iceberg.
Edit: more context on Ziz, http://zizians.info/
Edit: found this https://www.reddit.com/r/SneerClub/
As someone said over there, it's about time to start putting lithium in the Bay Area water.
Even if this is 2 orders of magnitude too high it's worth a lot of our time. But these are probably mainly normy researchers not AI safety-ists. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
With no power, comes no responsibility.
But of course, if ML is a person's pet interest, they will likely think we should increase spending on it. That doesn't make that a smart decision...
I’d argue that power is in relation to reach and the capacity to influence. In this regard, ML has quite a bit of power these days, from deciding whether you are allowed to take a loan, to driving engagement in social media, to predicting and influencing what you will watch, to the way our language is even used.
There are also quite a few implicit ways that ML models have influenced society, eg GPGPUs with emphasis on ML, and ML accelerators everywhere. The effectiveness of certain algorithms made us use them more often which influenced NVDA and Google towards developing hardware to accelerate those common cases thereby creating a feedback loop where our algorithms are chosen based on works well on our GPUs (cf transformers).
As such systems become more prevalent and influential, ensuring safety and explainability will help us prevent pitfalls that put humans at risk.
Personally I view it as no different to the hype over crypto or the fact we get a huge breakthrough in fusion every 3months but somehow never get any closer to an actual commercial reactor...
Meanwhile, we KNOW as a fact that climate change is going to be devastating. And there are numerous areas with nuclear wars brewing from far east to Europe.
I get that a super AI is a more interesting thing to worry about. But we need to deal with out actual, serious, issues first IMHO.
Power is not just about being more intelligent than us. An isolated airgaped super intelligence is powerless.
Because I don't consider myself sufficiently educated in these matters, I default to using GiveWell and their mosquito net recommendations as an effective low-effort approach to making monthly contributions to charity.
Overall, is there anything about this approach which is objectionable? How do philosophers go from something like GiveWell to something like fending off the AI apocalypse? Is this ideological expansion inevitable?
"All human lives are equal" then expands to "and there are more unborn lives than there are currently living lives, so it's more important to save unborn kids in 200 years by diverting AI catastrophe now".
I personally don't think unborn lives are worth anything, and as such I don't follow that jump. I don't feel a need to justify it either.
I think there's two additional easy ways to veer off the EA course.
The first is "how do we know EA is right about the charities?" The answer you'll get is basically "Sure, you can donate a kidney to your father and know it saved his life, you can verify that... but the people verifying your EA kidney donation does more good are smarter than you. They're rationalists. They do math. They know selling your kidney, which would have only saved one life, to a black market and getting enough money for 50 mosquito nets, which will save 20 lives, is the right deal. The spreadsheet says so."
I do not buy that the people running and choosing EA charities are actually the smartest people, especially when it comes to having a positive impact on lives. Sure, they talk a big talk, but I know for a fact that when I give money to a local shelter, the money gets used for good. I know for a fact that when I donated to a certain EA charity I won't name, the money went into "AI research" which is to say into the pockets of a rich white dude. I know for a fact that the average EA rationalist I have met does not come off as a caring human.
A second way that one can veer off the EA course is by deciding that one values some lives more than others. For example, if I live in a small city, I may decide I value the lives of orphans in this city more than the lives of the orphans in some third-world country. Sure, feeding an orphan in africa is cheaper so EA would say do that, but I may decide that the intangible value of improving my local community is worth more than "getting the most bang for my buck". I think this is also a totally reasonable argument, and in fact I would much rather live in a world where everyone tried to improve their local community than one where everyone used a spreadsheet to try and maximize their "best improvement per dollar".
I think I'd rather prioritize making the billions on earth we have right now very happy instead.
There are equally counter-intuitive consequences of discounting future lives or not valuing them at all. What makes this specific point in time so much more important than every other point in time? Can we do anything we want to make our lives better regardless of future consequences?
It's simply that writers like to write, forum commenters like to comment, philosophy academics like to publish papers, and there's only so many ways you can rehash The Life You Can Save.
Advocating for mosquito nets and praziquantel is like advocating that programmers should use version control - it may be true, but it's not going to drive traffic to your blog or impress people with how forward-thinking you are.
The people who haven't expanded their ideology in debatable directions aren't getting to the front page of HN.
If you disagree then why is OpenAI's safety committee talking about how you should need a government contract to buy an A100.
Fuck, really? I hadn't heard about this. If true, it's the godfather of all ladder pulls.
Not to mention, it's categorically awful to silo this stuff and grant it to an org that's so opaque. Good luck separating the wheat from the propaganda chaff without access to the tools that produce it.
Learn something new every day!
I got caught up by the multiple times the main character made sweeping assessments of things with insufficient information, and just happened to be right by accident (or author fiat I guess). The main characters figure things out by Thinking Really Hard, and experimentation is only ever done to prove they were right all along to other characters.
That's an arrogant character who keeps falling into the right answer, not a scientist.
When is Harry Potter and the Reproducibility Crisis coming out?
If it "goes above the head" of the people it's targeted at, that just makes it a poorly written puff piece.
I think that Eliezer is very well-situated by both age and social niche to find HP fanfiction vastly more appealing than most human beings do. I think that he genuinely finds the idea of "Harry Potter but more as a hard fantasy with a hero bent on breaking the setting by using everything without regard for genre convention" interesting as a story concept. And when he set out to write up his ideas about rationality in the form of something other than dry nonfiction essays, I think that HP fanfiction sounded good to him.
I don't think that he correctly found the medium that is maximally appealing to the potential target audience for his ideas (like, I definitely do not want to read a super long Harry Potter fanfic, and I think I'm joined in that by essentially everyone who's not a very geeky millennial), and I don't think that he gritted his teeth and wrote a HP fanfic in order to increase the audience for his ideas despite not liking HP fanfics.
This hits the nail on the head for me.
I mostly enjoyed the books as a kid, but didn’t really maintain being a Harry Potter “fan” into adulthood. Out of the dozen or so “hardcore” millennial nerds that I know really well, there’s exactly one person that I think would attempt to read something like that, and based on the reviews I’ve read, I would bet that she would struggle to finish it, if at all.
There is a somewhat interestingly vocal group of internet adults that insist that the default mode for nerds in their 30s is to be Harry Potter super fans, some of whom lapse. Like we were all raised Catholic and some stray.
I was a big fan of Animorphs at that age too, but you won’t see me going around reading or writing Hork-Bajir/Andalite fanfic. Because I enjoyed it as a kid and then went on to read other books.
https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_o...
it's an actual thing?
This is a joke, right? tongue in cheek or something. Or should I go get some Brawndo to water my basil...
HPMOR is really garbage-tier. It's parasitic on another series, doesn't have much in the way of a good story, and quite legitimately makes shit up. There are much better HP fanfics (and indeed books) written by people that are not as "smart" as Yudkowsky and also didn't have an agenda to push.
Judging by my participation in lesswrong and others, there are both feminists and sexist people, just like all other online forums. People are more free to be sexist online and every close knit online community has fair share of sexists.
" MIRI employee Jessica Taylor had a job that sometimes involved “imagining extreme AI torture scenarios,” as she described it in a post on LessWrong—the worst possible suffering AI might be able to inflict on people. At work, she says, she and a small team of researchers believed “we might make God, but we might mess up and destroy everything.” In 2017 she was hospitalized for three weeks with delusions that she was “intrinsically evil” and “had destroyed significant parts of the world with my demonic powers,”"
This is like what, one bad week away from AI Jonestown? At some point this isn't a community but a cult, and women usually tend to draw the shortest sticks in any cult environemnt with bizarre moneyed, male leader figures.
Look at the Ziz story - we're already there. (https://medium.com/@sefashapiro/a-community-warning-about-zi...)
However, this should not severely impugn the community as a whole.
https://forum.effectivealtruism.org/posts/7b9ZDTAYQY9k6FZHS/...
For some more context: https://rationalwiki.org/wiki/Scott_Alexander#Race_and_IQ
The generic terms "effective" and "altruism" are both very important terms, and this doesn't follow from combining them.
2. I like hearing rationalists describe “demon summoning is easy, Angel summoning is hard.” Genuinely curious about that.
The key point would be that it needs not be actively hostile to human values, but indifferent to them. Think of a Lovecraftian Elder God, not AI Hitler.
The supporting hypotheses would be that it's much easier to create an unaligned AI than an aligned one, as an aligned one is an unaligned one with additional constraints, and that unaligned AIs will wreak massive destruction.
Are riddles not OK for HN? Or should I have made it more clear that it is a riddle?
I know almost nothing about EAs but what I know makes me think that it is likely P(SA)>P(SA|EA)>0.
AI safety is a real issue, but look at what it looks like so far.
- "Machines should think, people should work". That's Marshall Brain's "Manna". It's also working at an an Amazon fulfillment center, where the computers tell the humans what to do.
- "Paperclip maximizer". A corporation is a paperclip maximizer. If you take Milton Friedman seriously, that's what a corporation is supposed to do. As I point out occasionally, most of the ethical problems foreseen for AIs are known ethical problems with corporations.
Those are the classic problems. As the technology advances, we're discovering new ones in the large language model space. Those issues are well known here.
Corporations are not capable of having the "fast takeoff" which is an integral part of the doomer narrative here. Furthermore, a corporation being a paperclip maximizer is an abstraction that obviously falls apart in the extremes, if for no other reason than because at some point destroying the world for the sake of more paperclips is going to have a negative ROI....
Meanwhile, the (AI) paperclip maximizer is actually trying to maximize paperclips and does not have the sort of internal checks that would apply to real-world orgs. That is the crux of the problem.
Elsewhere in the thread, it's brought up that AI could move faster than corporations generally move. That might be true, but there are still physical limits: the speed of light, moving mountains of steel takes energy, etc.
What do you mean by this?
At some point it had more soldiers and controlled a greater subject population than the government of Britain itself.
I'm not sure it was ever consistently profitable, though, which is the main paperclip corporations ostensibly maximize. Which itself speaks to the problem: superficial objective functions don't always determine outcomes or behavior.
That's key to the story of the paperclip maximizer - the paperclip maximizer will go about its task by trying to improve itself to be able to best solve the problem, and once it's improved enough it will decide that paperclips would be maximized by destroying humanity and would come up with a plan to achieve this outcome. However, humans may not realize that the AI is planning this until it's too late.
Individual corporations aren't capable of unbounded exponential growth because they can't keep the interests of the humans that make them up aligned with the "interests" of the corporation indefinitely. They develop cancer of the middle management and either die or settle into a comfortable steady-state monopoly.
Market systems as a whole can and do grow exponentially - and this makes them extremely dangerous. But they're not intelligent and so can't effectively resist when a world power decides to shorten the leash, as occasionally happens.
A human+ level AI would be able to understand and improve its own hardware and software in a way we can't with our own brains. As it improves itself, it will get compounding benefits from its own improvements.
https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
A poll of NLP researchers showed 37% agreed that catastrophic outcomes of AI comparable to nuclear war were plausible.
I find this kind of fascinating, really. Because of the way we sort work and create institutions, there's a structural hook that ties in the persistence of the problem (or the creation of a perception of the problem) with the longevity and motivation of the institution / entity that is attempting to solve it.
Yeah, you could contract out to a third party (which would be better), but you're still establishing a relationship where their future earnings are dependent on coming up with a certain result, unless everyone involved is incredibly self aware and conscientious and is willing to work to put themselves out of a job. Not impossible, but a rarity.
Take the DEA, for example. Every indicator points to deprecation as the way forward. The science says that treating drugs like a crime is ineffective and creates all sorts of externalities, and there isn't even broad public support for the War On Drugs. It's expensive, ineffective, dangerous, wasteful. And it keeps an absolute fuckton of people employed, at all levels. These people have a vested interest in maintaining the status quo, if for no other reason than their positions afford them power, quality of life, etc. So they lobby to keep stuff illegal and go on campaigns attempting to convince people that they'll peel themselves if they get high.
I'm not suggesting that EA and this AI thing is comparable to the DEA, mind you. If given the choice to keep one I'd be hanging out with you guys. That said, I'm ever leery of surveys in general, and there's a recursiveness to using your own data to back up your claims that doesn't sit well. If it's truly something that warrants money / time / attention, there's gotta be external sources that corroborate - Which I don't necessarily doubt, but that's not the thing we're looking at, here.
Granted it is a hard topic to discuss, buut jeez there are some ignorant comments here.
Isn't Nick Bostrom also a longtermist? It's probably the reason for the focus on AI. Also wasn't he one of the founders of Longtermism?
A ton of the article is about immoral behavior of men in positions of power towards women. And the misogyny.
> The underlying ideology valorizes extremes: seeking rational truth above all else, donating the most money and doing the utmost good for the most important reason. This way of thinking can lend an attractive clarity, but it can also provide cover for destructive or despicable behavior.
I think this quote really sums things up for me.
Places where AIs aren't used to command peoples lives will probably not even realise that an AI apocalypse had happened. Perhaps a case of western navel-gazing?
Whether people know or care about a risk doesn't mean that the risk won't affect them!
> I also doubt they would be - negatively - affected if a rogue AI would turn out the lights in the electronicified world.
A rogue AI might as well just kill everyone rather than just the Western naval-gazers, as this would bring to 0 the possibility that a human could ever turn it off / thwart its plans.
How would it do that?
As one example it could design, synthesize and spread a highly contagious and lethal pathogen.
I always wonder why these tribes haven't yet invented the iPhone? Obviously since they don't seem to need it.
For some reason you seem to expect a civilization with the technological sophistication of globalized society with 10,000+ years and billions of people and countless global resources fueling them to suddenly emerge in the small geographies and populations of uncontacted tribes? If you were trying to be witty, you came up short.
Edit: or were you trying to play off "negatively affected by the modern world" as, "well they don't have iPhones so they don't have the Social Dilemma, everything's peachy" instead of the blatantly obvious "we destroyed their world and they had no clue" angle...
I mean if everybody has a super-smart personal assistant in their home, who thinks independently, why would they need or want a ruling class, when we can simply have democratic socialism?
You can also look at it this way. The only real objection against socialist societies was that people are lazy, and don't care enough about producing stuff or resolving economic inefficiencies. But with AI, this objection is gone. Nobody really wants to be in a constant "resource competition" or whatever, it's an effort. People generally want to have fun and enjoy life.
In theory, yes. In practice, the circle has been continually expanding with technological progress.
Stated differently: What's the point of being powerful if you cannot rule over other people? What's the point of having thousands of empty palaces, when no other humans see them or care?
Only if by continually you mean in sum. The holocaust and the genocide that have come since were pretty huge back slides and the tran Atlantic slave trade was a fairly new innovation in scale and kind even if slavery had existed before. Both of those seem if anything enabled by technology. And certainly the rich and powerful are unlikely to to kill absolutely everyone else, but I thought earlier you were arguing that AI would see the end of the rich and powerful. Now you're just saying they'll want to have pets
[0]: i.e. the system in the US and western Europe that prevailed over the USSR at the end of the cold war, technically not "real" capitalism.
That's my point! Your personal AI (in collaboration with other AIs) can take care of figuring that out as needed. You don't need capitalism anymore.
They have a business model problem. A century ago, recorded music obsoleted low-level musicians. Over the last two decades, the World Wide Web obsoleted most journalists and printers. Artists now face a similar problem.
An insight from ChatGPT is that "creative" is easy, but "accurate" is hard.
* Founding and scaling GiveWell, synthesizing and analyzing global poverty research to make recommendations. This has helped guide about $1B+ in really valuable work, the largest portion in anti-malarial bednets. https://www.givewell.org/about/impact
* Convincing Dustin Moskovitz and Cari Tuna to allocate their giving along these lines, building Open Philanthropy which has made a lot of grants that look really valuable https://www.openphilanthropy.org/grants (this is in addition to the ones where they granted at GiveWell's recommendation -- don't want to double count).
* Michael Kremer won a Nobel prize in econ (shared with Esther Duflo and Abhijit Banerjee who I'd describe as EA-sympathetic but probably don't consider themselves EAs?) for global poverty research.
* Built an "AI Safety" subfield of computer science, with the 2016 paper "Concrete problems in AI safety" having 1.8k citations.
* Founding and scaling 80,000 Hours, helping people figure out how to use their careers to do more good. This includes looking into potential career opportunities, writing up public advice, and 1:1 advising, and 1k+ people have cited them as part of why they changed what they were working on. https://80000hours.org
(Not especially representative; there's been thousands of people working along these lines for years and I'm sure I'm missing a lot of valuable work.)
[0] https://forum.effectivealtruism.org/posts/xof7iFB3uh8Kc53bG/...
[1]https://www.forbes.com/sites/sarahemerson/2023/02/28/effecti...
[2]https://intelligence.org/wp-content/uploads/2022/12/Independ...
I just skimmed the report you link and nothing jumped out at me as matching this -- could you be more specific?
I'm convinced most EAs are psychopaths.
With sufficient future sight and the power of the exponent > 1, one can justify many present actions. Sadly, the EAs are small thinkers.
True grand scale actions require truly grand visions https://cosmicbraintrust.org/
https://sinceriously.fyi/brent-dill-confessions-full-redacte...
This entire social milieu is a goddamn horror show.
It encourages mortal men to play God at levels of complexity that they can't possibly comprehend.
It should be banned for the greater good
>> “earning to give,” the idea that people like Bankman-Fried should do whatever it takes to make a lot of money so they can give it away. To amass his billions, Bankman-Fried allegedly defrauded his customers, and critics have said his downfall shows that EA is vulnerable to a myopia that allows the ends to justify illegal means. Among EAs themselves, however, the most salient criticism is subtler: that living at the logical extremes of the ideology is impractical and a recipe for misery.
In buddhism https://en.wikipedia.org/wiki/Middle_Way middle way is advised for this very reason
There is no canonical way the aggregate utility between persons, and people probably don’t actually have utility functions. And, there are moral laws which should have at the least some weight when weighed against a “but this action seems to have better outcomes overall, on average”, and possibly infinite weight when weighed against such things.
Nevertheless.