The Reluctant Prophet of Effective Altruism
newyorker.com
newyorker.com
(It's also one I took a long time to come around to. For years I agreed in theory that trying to prevent civilizational catastrophe could be the most important thing to do, but didn't agree that there were tractable projects in this area. After deciding that there are, however, I quit my 'earning to give' job to go work on trying to catch future pandemics earlier.)
It's... actually kind of not, which is why this is so upsetting. EA seemed to have a handle on a way to get libertarian-leaning, economically privileged, inward-focused men (and, yes, that's the demographic almost to a person) to think about ways to help people they'd never otherwise have given a passing thought to.
Then roll the clock forward a few years and they're all back to sci-fi fandom arguments. This nonsense is exactly what you would have expected from these people. The notable part is that for one shining moment they pretended otherwise.
One good list of projects: https://forum.effectivealtruism.org/posts/u5JesqQ3jdLENXBtB/...
In any case, my 'alpha' was something like, I'm a software engineer and engineering manager willing to work in an academic environment well below market wages, in a situation where the organization needed someone with my skills.
The main problem I see with AI risk is not that it is a small problem, or that there are too many people working on it, but that we don't really know how to make progress and historical attempts maybe have ended up making things worse.
Still, I think gradys' point above is a good one: the fraction of people working on making AI more capable is far far larger than the fraction working on making sure it reliably does what we want it to do even in unusual situations.
The article ends with MacAskill saying: “My No. 1 worry is: what if we’re focussed on entirely the wrong things?” The problem with too-longtermism is it’s impossible to know the context of these problems outside of a decade or two. For pandemics, no one thought we’d have a vaccine so fast. Perhaps in 20 years custom vaccines will come on the market in days. It’s impossible to know, so let the next generation figure it out.
Where you trying to add a point? Your comment reads like a non-sequiturs over gender inequality that goes nowhere fast, to the point that it reads like something generated by a machine learning model in training.
https://forum.effectivealtruism.org/posts/ThdR8FzcfA8wckTJi/...
Why write angry rants about EA people? There's the broader 99%+ of people who don't donate to whatever you deem more worthy. In all likelihood, you're in that 99%--there's probably income you could spare to give.
I mean, I'm worried about climate change, but not because it's fun to worry about. And I put a lot of my time and energy into thinking about how to do something about it effectively, but not so much because it's fun. Sometimes I'll hit on a fun idea in the process, but I'd be much happier if I didn't have to worry about it at all, but that's not really a choice I have.
I'm not sure how much I should worry about AI, but I'm certain that for the people who are worried about it, it's because they're convinced it's a threat, and they aren't having fun.
I've met a lot of these people and regardless of how you feel about the sincerity of their convictions, they are definitely having fun.
> On the morning of September 12, 1933, on a miserable, wet, quintessentially English autumn day, at the intersection where Russell Square meets Southampton Row, Leó Szilárd waited irritably at a traffic light waiting for it to change from red to green. He had just attended a lecture by the great English physicist Ernest Rutherford. Rutherford, known to many as the father of nuclear physics, was discussing the newly prophesied release of energy from atoms, most notably by science-fiction pioneer H G Wells in his book The World Set Free. In his baritone voice, Rutherford, acknowledged master of the atomic domain, dismissed this fanciful idea as nonsense. Any thought of releasing the energy locked in atoms, he said, was "moonshine".
...
> What Szilard realised as he stepped off that curb was that if we found an element that when bombarded by one neutron would release two neutrons, it could lead to a chain reaction that could possibly release vast amounts of energy
https://blogs.scientificamerican.com/the-curious-wavefunctio...
Secondly, you can have a diverse portfolio of causes you care about, and Effective Altruism has recommendations for areas including global health and animal welfare. Taking AI risk seriously doesn't mean you can't give time and money to other causes with more tangible results.
But if you've spent the past decade arguing precisely the opposite - that instead of spreading philanthropic dollars over broad portfolios because true efficacy is unmeasurable, they should be directed to causes which provide hard evidence that they're among the most efficient use of a marginal dollar, and that charities that aren't conducting RCTs to prove how much better off people are after being fed aren't really evidence-based enough to recommend - it's going to raise eyebrows when you pivot to recommending people also give their money to opaque organisations with vague goals and zero evidence they can achieve them, whose only thing in common with the highly evidence-focused aid agencies is that their pitch is nerdier than the average nonprofit. I mean, there's nothing particularly wrong with "I give to causes because their pitch speaks to me" as a heuristic for philanthropy - it's how most altruists think - but it's not how EA pitched itself as being different from that and caring about efficacy and evidence.
The Wells analogy is awkward for the AI safety crowd for another reason too. Wells' central thesis that an atomic super weapon would guarantee a peaceful, utopian world turned out to be horribly wrong. We got actually used bombs, arms races, proxy wars and very nearly used bombs instead, and that's without Nazi atomic scientists getting a heads up. To the extent his writings on the subject of atomic weapon safety had any effect at all on atomic weapons, they made them more imminent and more likely to be used. If marginal dollars spent on promoting the distribution of The World Set Free would have had any effect at all on the probability of armageddon, they would probably have increased it.
On your first point, I can donate towards anti-malarial interventions and towards reducing AI risk: two very diverse causes, but both under the EA umbrella. Diversifying amongst recommended EA causes seems to me a good bet, especially given I expect some of them to require pivoting away from as more research is done.
Perhaps we can summarise the last decade of EA into something like: "given the available information (largely based on RCTs, research and an understanding of moral philosophy), we expect that money spent here will do more good than money spent there. More information will change this expectation."
We now have information that there is a potentially very grave risk confronting us from AI. For other cause areas we can use evidence like randomised controlled trials to inform our actions. In the case of AI risk, this type of evidence is unavailable, but there are nonetheless other types of information available to us that suggest it is worth taking seriously. This information takes the form of compelling arguments from credible people in the AI industry and the industry's achievements to date. Of course we'd like to have firmer footing, but the nature of this risk doesn't permit it. We can either ignore it, and potentially walk to our doom, or we can take it seriously and try to reduce the uncertainty and the risks.
I agree with you that this raises eyebrows, but I don't think it's a gross departure from the last decade.
I think it is wrong to say we have "zero evidence" that organisations committed to reducing AI risk can achieve their goals. I think it is more accurate to say it is highly uncertain that they will be able to achieve their goals. My view is that we should try to address AI risk because the stakes are high.
I would have sided with Albert Einstein in trying to renounce nuclear weapons, even though we have "zero evidence" for humanity ever coming together to do such a thing in the past.
Of course - nuclear weapons really do exist and AI is only a hypothetical threat at the moment. My point is that there was "zero evidence" that the world could come together to voluntarily control a technology, and yet people like Einstein tried to make it happen, and with good reason. Dismissing this is like taking Rutherford's view in the 1930s and saying there's very little point worrying about nuclear weapons because they're not possible, and it's even more pointless thinking about how we might control them.
Finally - why might the pitch speak to nerds in particular? I think in part because it is based on Bayesian reasoning, which is both nerdy and incredibly effective in situations of uncertainty.
There's nothing unique to EA in this (people convinced that most aid money is wasted sometimes give to organisations they have some sort of ethnic/political/religious affiliation or people they generally like unconditionally, and sometimes that trust heuristic has even been right!) but it did purport to be the school of altruism that didn't.
As for the Wells analogy, the two most notable uses of AI safety funds seem to be (i) attempting to build AIs and (ii) distributing fiction they believe best illustrates the possibilities of AGI, so I'm not sure I'm altogether off the mark in imagining what well-intentioned utilitarian nerds of the time (Wells certainly was one) might have done with more cash. If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal for a class of technologies which evolved self awareness from being rewarded for meeting human needs, then popularising the idea of conflict between human and AI could be more likely to have extremely adverse unintended consequences than intended ones. Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.
Can you let me know more about that?
> If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal
I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want.
> Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.
It sounds like you think AI is risky too - even if only at this intermediate level? I'm curious - do you have a view on how we could go about reducing this risk?
Compare the level of transparency and need for funds of OpenAI LP with the average organisation criticised by GiveWell for not being transparent about how they spend the money, not providing any metrics or trials, and/or having more money than they need to fulfil their short term goals and plenty of access to more.
> I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want.
Is "people might not recognise how the second order effects of pursuing a goal adversely impact humanity" a problem unique to decisions made by AI? If anything, I would have thought that in addition to actually existing, independent human intelligences' resource demands were already more competitive with human needs than a silicon-based AGI's is particularly likely to be, and demonstrably neglectful of second order effects. With those problems mostly unsolved, it's hard to see tackling the same thorny problem but with hypothetical goals of hypothetical entities would be seen as the more impactful line of research (as interesting as it undoubtedly is). Nothing wrong with intellectually curious altruism, but if you're describing yourselves as the arbiters of effectiveness...
> I'm curious - do you have a view on how we could go about reducing this risk?
I don't have an easy solution to a range of complex mostly political problems, no, and certainly wouldn't argue for investing in studying them on the grounds of other philanthropy being less effective. Generally attempts to impose bans and developing more, better weapons quicker seem to have been attempted solutions to non-AI based arms races and neither of them have a great track record. Misplaced faith in the output of current techniques we call AI as superior reasoning free from human bias rather than garbage in garbage out mathematical transformation skewed by massive unseen flaws in training sets and prone to non-obvious bugs isn't likely to be reduced by talking about allying with self-updating superintelligences...
Isn't it substantially worse than this? The EA people generally are, or are closely tied to a lot of the AI risk people, right?
Similarly, a large fraction of people at the Nucleic Acid Observatory (where I now work) are EAs, for similar reasons: becoming convinced that this is what most needed doing.
So many words just to say "they don't value the exact same stuff I value".
You're free to put your money where your heart is, just like them. It's not right to hit back at those who are doing what's right by their own account.
This argument is appearing again and again in this subthread. "Putting your money where your heart is" is the exact opposite of what EA claims to believe. EA is about investing effort and money into areas with the highest marginal value to humanity.
So if they're spending their time talking about the robot invasion, we can only conclude that's where they think that effort should be spent.
Obviously they're free to do whatever they want with their own time and money. But if they spend that time pretending that SkyNet or WOPR is the greatest threat to humanity that the rest of us should be spending our marginal effort on, it's on the rest of us to call them out on that nosense.
And it's the individual who determines that? Or does your own personal opinion override everyone else's world views?
[1] "Responsibitility", i.e. not just my "right" in a liberty sense! Again, the theory behind EA is that we should collectively decide on the most effective altruistic actions based on rationalist principles. Those principles absolutely include "you're wrong about the robot overlords thing".
I think it's nonsense because the ostensible core purpose of this movement is about taking competent, motivated people and deploying them in such a way that they can make a meaningful change in how the world works. However, the main effect of their recommendation is to go out and participate in the system in a pretty standard way. How will meaningful change happen in a system in which you aren't acting in any way differently to the standard operations of the system?
Sure, you donated a significant fraction of your income to some non-profit. But that effect pales in comparison to the economic contribution of your labor in big tech/finance/whatever which are the dominant institutions structuring our society. And most of the non-development problems such as AI risk or nuclear risk come from the same kinds of structure dynamics of society that caused the wealth inequality and environmental degradation.
So I believe, for a group supposedly run by very smart people with very noble ideals, they kind of fall pretty short of where they could be and don't go deep enough. They are proposing band-aids to problems rather than true cures. To me it just becomes another part of the non-profit industrial complex, a neoliberal-shaped industry that effectively perpetuates the problems it purports to solve since it relies on the existence of those problems to survive. We've known for a long time how charity and aid can do more harm than good and plenty of strategies formerly championed by 80000 hours have now been found to be more extractive/exploitative than helpful, such as microlending.
Fundamentally, they just can't seem to make the necessary leap to criticising capitalism and strategising against it, instead promoting it as the preferred tool to use. But how will capitalist tools solve the problems caused by capitalism? If instead they got all their smart followers to use their privileged positions and training not to reinforce the most powerful institutions of capital but instead build alternatives, I would think they are on the right track, even if it included AI risk initiatives.
I don't think 80,000 Hours has ever promoted microlending. They were critical of it ten years ago: https://80000hours.org/2012/11/is-microcredit-mostly-hype/
For the record, I don't think people like Musk, or, say, Andreesen or Altman are EAs themselves. But they are the kings picking court jesters like Yudkowsky who then have the resources to (re)define the movement.
(I would also say his influence has decreased a lot over that time, as he is writing much less now, and there are a lot of other voices on AI risk.)
It makes sense. We know how to deworm. That is an execution problem. The ones I find more interesting are in planning stages, such as how to we fix or supersede capitalism, how do we mitigate climate change, how do we persuade people in power those things need to be done before the problem is so big it becomes intractable.
Ajross, you don't seem to really believe what you are saying. Sci-fi arguments? Really? Since when is the risk from nuclear war, pandemics or artificial intelligence constrained to sci-fi? The Spanish flu, Black Death and the Plague. The Cold War. These events should all inform our way of thinking about risk.
The smartest people I know are communists, but I imagine the end state of both is pretty similar - everybody enjoying a surplus economy where work is optional and done for the betterment of ourselves and others.
I don't know your specific reasons for thinking it nonesense, but as someone who has read up on AI risk and consumes a lot of sci-fi the resemblance is very superficial.
Different people, growing in different environments will want different things. As long as our wishes clash, it will cause fighting along the way.
Also poverty is kinda a feature. You need worst performing people to be put in negative display.
Obvious but hated fact: the human race is nowhere as mature as it thinks it is, and until such time that the adult man-babies and woman-babies are identified, called out, and removed from positions of power we are helpless to their immature wealth and power driven stupid fantasies. The immature rule the world, even if they don't sit in positions of power, because the immature demonstrate their attitudes so loudly they overwhelm all public discourse.
Yes, existential risk of general AI might be small; but even fewer people are working on it: only about one hundred, worldwide. Similarly, the reason E.A. stepped off the climate change train is because it was a popular issue, which means the marginal benefit of one person contributing decreases.
When more people direct attention to AI safety, another area will be the one where E.A. can contribute most, and the focus will shift.
You can make that argument, but I think the median view among people working on AI alignment is that the risk is quite high, in the range of >20% chance of disaster within their lifetime.
Still, the changes to society are absolutely hard to grasp. Humans may face (economical) obsolescence in a few decades[1] (humans need not apply[2]). That puts us in a delicate situation: if humans are economically obsolete, a society that only values economic value (and not consciousness per se) will have all reasons to slowly but surely get rid of humans (and animals).
We need a revolution of meaning and ethical understanding: the understanding of the fundamental value of consciousness, and to structure our lives and all institutions around it (each individual needs to have a good understanding of such ethics and meaning so that society as a whole is robust). I think formalization of ethics (which I'm a proponent of) and progress in metaphysics is going to contribute as well to this end. I think EA is simply part of this new understanding of ourselves and the Universe, a sort of coming out of a dark age -- and there's the opportunity to create a lasting society amazing for everyone.
[1] My personal guess-estimate is about 50(-20/+40) years to complete obsolescence, although various researchers have wildly different estimates.
The problem with Evil Boltzmann Brain is that it could destroy all life in all galaxies through the infinitesimal chance that it spontaneously produces space travel.
Therefore, I have dedicated my entire life to fighting EBB instead. Currently, this requires that I buy an iPhone with the money I had planned to use on GiveWell.org but it's all in the plan.
I have not forgotten the goal, though. If I consume all resources on Earth, then I can stop evil AI at the same time as I increase entropy and reduce the possibility that EBB can utilize any energy differential. This requires that I consume consume consume. Which I shall do out of altruistic purposes.
My thought was that if you're going to assign infinite weight to unlikely possibilities, then why not devote your entire life to preventing time travel from being invented in the future?
Someone could invent a time machine and destroy the Earth before humanity had a chance to exist, etc.
It's just about as falsifiable as what they are working on.
---
Also, if you're going to be so abstract about it, then why you wouldn't want to optimize for the happiness and future vigor of AIs? If you're going to be a vegan, then presumably you care about non- human consciousness.
So what's wrong with a future with flourishing AI and without humanity?
There are too many ridiculous things to count here, and normally I would ignore it, but it seems like people are taking it seriously and it has consequence? (And the article is pretty well written either way)
---
It's weird that he had doubt that's like 1 sentence in the article, and then entire lifetimes of work that simply ignore the doubt:
My No. 1 worry is: what if we’re focussed on entirely the wrong things?” he said. “What if we’re just wrong? What if A.I. is just a distraction?
And as alluded to in the article, the wild swing from helping the poor to preventing AI has a huge moral hazard. With the former you can course-correct your actions based on observation of outcomes -- what happened as a result of the donations?
With the latter you can justify any action today by just pushing out the hypothetical benefits further and further into the future.
It resembles a religion that collects money today, promising benefits in the afterlife.
Not necessarily anything, but a pretty mainstream view now is that dangerous AIs need not be conscious (and indeed, that AIs can still be dangerous even if it's somehow impossible for an AI to be conscious). So you could imagine futures in which an AI conquers the world but has no experiences and can't either flourish or fail to flourish.
A computer virus is a familiar existing example of software that can be dangerous but that could probably not be said to flourish, even if it spreads widely, accomplishes everything it's programmed to do, and persists indefinitely. So we could envision dangerous AIs that are like today's computer viruses but that are also programmed to cause some kind of effect in the outside world, and that have some ability to learn and adapt so that it's difficult to prevent the effect and difficult to remove or deactivate the virus. That could be pretty dangerous to have active anywhere on the Internet, but still probably couldn't be said to flourish.
I struggle with assigning infinite weight and unlikely possibilities. But we can stay in the every day world and ask what credence do we place on the time machine possibility, and why? All information I have read suggests that it is essentially impossible, and so effectively I don't worry about this contingency at all. And then, is there anything we could do to prevent the time machine scenario? I can't think of anything at all.
On the other hand, to focus on one particular risk, there are compelling arguments made by experts in the field of artificial intelligence (e.g. Stuart Russell) that say that we face real risks based on the current trajectory of the technology. Is there anything we can do to reduce the risks? That seems like an important question.
It is important to be sceptical, and whilst I don't place 100% credence on the likelihood of a truly catastrophic AI outcome, I do think it is worth taking this possibility very seriously. There is a strong chain of reasoning behind the claims, and they are worth engaging with, especially given the fact that we are in a position to influence the outcome.
> if you're going to be so abstract about it... what's wrong with a future with flourishing AI and without humanity?
We can park the abstract side for a moment and think concretely about such a future. I want humanity to flourish, particularly during my lifetime. Credible people are currently ringing alarm bells about the potential destructive power of a technology we are actively developing, and so I think on balance it is better (at a minimum) to listen and be vigilant than to dismiss it. Better yet, I think we should engage with the arguments and see if there is anything we can do to reduce the risk of a potential bad outcome.
To the extent that these concerns are not overblown, one way to avoid the dangers of advancing ML would be to stop all the effective altruists [1] in the field haring along at building it irresponsibly.
The people preaching about the potential future dangers of AGI are the same ones pouring money into ML; if they actually meant what they said, they'd pour that money into their opponents.
The current practical dangers of ML are concerned with enforcing social inequalities and concentrating power in the hands of unethical actors. The fact that many of the unethical actors claim utopian reasons for their reckless disregard of consequences is not a reason to believe them.
[1] https://twitter.com/emilymbender/status/1556691543850831872
To me it almost just seems like a "different people can have different values" sort of thing. If you want to value current lives more than future lives, you'll just come to different conclusions than someone who believes otherwise, and maybe both sets of conclusions are okay.
I definitely agree with your second paragraph, however, and people who value current lives more than future lives will often come to different philosophical conclusions. It's interesting, however, that this frequently doesn't cash out into different beliefs about what we should do today. For example, I think the value of work on pandemic prevention or AI alignment does not depend on multiplying by zillions of potential future people: they're worth doing even just for people alive today.
Ex: https://forum.effectivealtruism.org/posts/KDjEogAqWNTdddF9g/... :
Let me rephrase this in a deliberately inflammatory way: if you're under ~50, unaligned AI might kill you and everyone you know. Not your great-great-(...)-great-grandchildren in the year 30,000 AD. Not even your children. You and everyone you know. As a pitch to get people to care about something, this is a pretty strong one.
But right now, a lot of EA discussion about this goes through an argument that starts with "did you know you might want to assign your descendants in the year 30,000 AD exactly equal moral value to yourself? Did you know that maybe you should care about their problems exactly as much as you care about global warming and other problems happening today?"
Seven billion (or even a couple of million) humans in the world today are obviously a sufficient rationale to pay attention to existential threats or stuff that could blow up in our faces including pandemics, climate changes and AI systems, but proponents of actions with apparently vanishingly small probabilities of improving safety tend to need to invoke gazillions to justify prioritising them over tangible solutions to the problems of millions in Bayesian/utilitarian framework.
Trying to sidestep common arguments about the personalities, but I guess an example would be an ideal-version-of-Bill-Gates, and an ideal-version-of-Elon-Musk. Gates Foundation mission statement seems more weighted to current lives, while Musk is more oriented toward multi-planetary existence of future generations. Still some overlap though, Musk's can also be argued to be relevant for current generations.
There is always a risk your company doesn't become the next unicorn.
There is always a risk your society fails to grow exponentially for two thousand years. Our current track record is that exponential growth happens until it stops, which is pretty bad news for anyone whose math assumes that exponential growth never takes a break.
In the same spirit, I might as well claim my after-meal $2 ice cream relieved me from hunger.
It's hilarious how much this sounds, and the science-fiction writer Charles Stross has pointed this out in the past, Russian cosmism [1], a weird offshoot of Christianity dressed in secularism and rationalism. That something like this emerged so quickly out of the decidedly pragmatic and genuinely secular altruism of Singer is in my opinion evidence of the weakness of Effective Altruism as a philosophical system. It's such a bureaucratic, mechanical system that it's already started to branch off into weirder, less utilitarian schools.
[1]https://en.wikipedia.org/wiki/Nikolai_Fyodorovich_Fyodorov#P...
Let's turn this into a trolley experiment, 1 million planets colonized with 8 billion humans each living the same lives we live today vs 10^58 humans of which the majority don't exist and are just made up. The extreme utilitarianists and long termists will pick the 10^58 fake humans even if there is a near 99% chance that none of the humans actually exist. That 1% chance would by far outweigh any lesser goal despite the fact that it is unlikely to happen in your universe.
Another problematic aspect is that if there were actually 10^58 flesh and blood humans, there wouldn't be any planets left for potential alien neighbours, by aliens I do not necessarily mean independently evolved intelligence, it could simply be a new human space race that develops in some quiet corner of the galaxy isolated from everyone else.
- no country was ever made wealthy by just giving them things. Wealth comes from passing through the difficult stages of development oneself
- you personally should do the good things you want to do in the world rather than just donating money for someone else to do it on your behalf
- Peter Singer uses the word "should" too much, and it's almost offensive. His philosophy is an empty philosophy pushed through by people who believe we are no different to apes. The older religions do this all much more justice in a far more thoughtful way.
I would suggest that people who want to make a difference with money do so on a local level. If they can create a local eden then perhaps someone else will notice and create another one locally where they are. It's kind of like small scale experiments which you can test and the ones that work will naturally be extended for little or no extra money.
It's just that this is (most likely) not the most effective way to make the world better.
And if your thesis is that this is the most effective way, because of copying etc, it would be useful to run some controlled experiments.
& govt. A poor family on section 8 vouchers, couple of kids in school, medicaid healthcare costs the government perhaps $100k/yr, + charities for food, clothing etc. You'll never get that in donations.
Which in most cases is going to involve destroying the existing institutions in societies that aren't creating much wealth, since it's precisely their existing institutions--corrupt governments and low social trust--that are preventing wealth from being created.
In terms of people setting up a successful and prosperous society and culture, then yes that can't be moneyed into existence.
Doesn't mean that thoughtful money can't help people or their society though.
We can talk about difficult stages at the individual level, but if an entire population faces things like dying from mosquito bites, I don’t think we gain anything or do anyone a service by not helping.
These are people who can’t pull themselves up by the bootstraps because they’re a little too busy trying not to die.
Edit: you mention not liking the word “should” in this context, yet you say people should do the good deeds themselves rather than donate money. Can you articulate why without a should here?
The point of donating money is that if you donate it well, it can have a greater impact than if you were to act independently. People have put a lot of thought and organization into it, and it isn’t an afterthought spurred by reluctance or laziness.
I believe the philosophical goal is to maximize the reduction of unnecessary suffering. I could be wrong, but with that in mind it could make a lot of sense to donate money rather than act independently.
Agree on the small problems that can be solved, but again i maintain that everyone needs to work through these stages themselves imo. People can help but ultimately the biggest help is opening up to useful trade, and enabling the empowerment of people from the bottom up.
It's a complex issue with no method without downsides.
Presumably, if people are spending less time taking care of their family dying of malaria, they'd have more time to create wealth.
Why?
If, for sentimental reasons, you get a kick out of working in the soup kitchen, by all means, work in the soup kitchen. Just be aware that you are doing this for your own entertainment.
> Wealth comes from passing through the difficult stages of development oneself
Is having people go blind from preventable causes or children growing up stunted supposed to build character?
> you personally should do the good things you want to do in the world rather than just donating money for someone else to do it on your behalf
The point is to do good, not feel good. Specialization is the very basis of human civilization. Software engineers donating their surplus to fund salaries for trained full time teachers is going to be a lot more effective than the alternative you're suggesting.
Once a much bigger player from the outside comes in and gives stuff for "free", like it has happened with some of the Western aid, then that kid's parents I mentioned at the start of the comment risk losing their jobs. After all, it's not worth it anymore for their employer to keep them, well, employed, cause the market that he's targeting is getting dumped by all the free stuff coming in via Western "aid".
These are, in retrospect, basic economics-related things, but which nevertheless took a while to sink in into some people's minds in the West (some are still oblivious to it all). I remember a book written by an African lady (I think she was from Botswana or from that general area), former worker at a big financial institution in the West, and imploring the West to stop "aiding" the African countries if they want to see them actually develop.
I would argue you want to do the latter because both net and medicine manufacturing are capital intensive - a small number of people are employed despite the large cost of setting up a factory. Most of the local economic contribution is going to come from hiring people to handle distribution.
Let's take an example mentioned in the article, Sam Bankman-Fried. After hearing about EA in college, he decided to earn as much money as he could first in traditional finance and then in crypto, so that he would be able to fund work he thought was important that was currently not happening. He was very successful at this, and currently has about $12B he's working on donating.
I think it's a pretty difficult claim to support that he would have furthered his goals more by picking one of the things he thinks needs to happen and working on it?
On the other hand, a lot of early effective altruist writing is from a time when there was very little money available for some of the most valuable things, a time when even "go work as a regular software engineer and donate a large percentage" beat directly working on valuable things for a lot of people. If you look at that writing now, however, since it is assuming a very different funding environment it looks like it sets the bar way too low for how much money you need to be able to earn before earning to give is likely your best option.
What happened to the Golden Rule? Or to Sartre’s “in choosing for himself, [each man] chooses for all men”? Surely if EA leads some to causing such wide-scale short-term pain in its name, at the very least it cannot be a humanism.
I realize this isn’t a strong argument, but when I hear about an EA champion, there’s often this dynamic where how they made money doesn’t matter. They’re wealthy, so clearly what they do is valued by the market and we should ignore externalities that aren’t priced into their income. It’s like Big Oil pointing to specific green outcomes that they fund; these outcomes are quantifiable yet funded by hard to price but reprehensible practices.
It all feels like Catholic indulgences… Maybe this is still a net-positive for society in that it provides people who wouldn’t be humanist anyway (e.g. libertarians) with a well-intentioned philosophy?
Yes. Initially he worked in regular finance, at Jane Street, and then crypto, founding the Alameda Research trading firm and FTX exchange.
> By definition this would mean capturing real money from retail investors, right?
I'm not very knowledgeable about crypto, but I don't understand how the definition of crypto means this?
The article we're commenting on describes one of the early successes of Alameda, where they saw that Bitcoin was consistently trading for much higher prices in Japan than in the rest of the world. In normal finance you mostly don't see this sort of thing for very long, because global markets are highly interconnected, but in this case it was logistically difficult to sell Bitcoin in Japan. In fixing this, Alameda lowered the price at which Japanese investors were able to buy Bitcoin, and raised the price at which people elsewhere were able to sell Bitcoin, keeping a fraction of the difference for themselves; I don't see how Alameda made others worse off?
With FTX they're making money by providing a platform that makes it easier for people to trade with each others. A bit of, 'in a gold rush cell shovels'. Again, I don't see how they're making others worse off?
> when I hear about an EA champion, there’s often this dynamic where how they made money doesn’t matter.
Sort of! If someone is initially not altruistic and accumulates a bunch of money doing something questionable, then if they later come around to doing good things with their money I'm going to be happy about that. So while I am not a fan of, say, Microsoft's business practices in the '90s which made Gates a billionaire, I do think his decision to spend his fortune the way he has is a good one, and one that should be commended. The alternative is that, like most other billionaires, the money goes to houses, yachts, and other things which are far less positive for the world.
On the other hand, SBF is in a different category: he was trying to be an effective altruist from the beginning. So if he did something massively harmful to earn his money (which, as discussed above, I don't think he did) we would need to consider that in figuring out his net impact.
Additionally, it is pretty common for EAs to argue that you shouldn't do harmful things even if they are majorly profitable and allow you to do a lot of good through your donations.
> I'm not very knowledgeable about crypto, but I don't understand how the definition of crypto means this?
Crypto doesn’t really power real-life use cases outside of trading crypto assets and associated financial services. For real money to come into the space there needs to be an influx from outside. Usually that means VCs and, more largely, retail.
I’m not necessarily arguing here that the crypto space is a drain on society, as I think that’s contentious, but the claim that most real money in crypto originates from people outside of it should be a truism. To me this makes SBF’s approach something misguided, as they’re reallocating capital from people at large to a single person. Even with the best intentions I can’t see this as a humanist approach. Some charitable projects do require massive funding and opinionated goals, but isn’t that what nonprofits and politics are for?
I guess it does come down to whether someone sees making money off of crypto as having strong negative externalities or not. If SBF was simply creating value by democratizing access then it’s just a regular job. If he was propping a scam industry that drains retail investors of real money then it’s a net negative even with EA as the goal.
Still, the inherent long-termism within EA seems to make it easier for people to embrace very Kantian views that maybe discount their short-term impacts.
No. What kinds of weird definitions are you using?
(You might be right in practice, I don't know. But you are not right by definition.)
> It all feels like Catholic indulgences…
And that is wrong somehow?
If you grant the assumption that the Catholic church was a worthwhile organisation doing good work, then the indulgences seemed like a perfectly good idea to me on a practical real-world level. (No clue about the intricacy of Christian doctrine, so I can't say whether they were kosher from a theological point of view.)
I disagree with what sbf is doing wrt political donations.
It is kind of difficult to grasp but when you see a homeless man, the best thing you can do to him isn't giving away money, it is hiring him and giving him a job that pays enough to pay the bills. When you donate money to the poor, you are basically subsidizing poverty.
Honestly Keynes already had the right idea, just give them a job to build commercially useless things like affordable housing.
Nah, people just are rather clueless on the what and the why of effective altruism.
The top causes are things like deworming, anti-malaria and AI risk. These are things that substantially increase economic productivity (the latter on account of people not being dead).
The problem is in the assumption that paid corporate work has zero social value. This is untrue. It has negative social value. If a rich person or a corporation that operates on rich peoples' behalf is paying you to do something, the odds are high that its total social value is negative, especially because we live within a moribund economic system that turns to extreme violence--fascism is capitalism in decay--whenever it hits the skids and people start to question it.
It's certainly plausible that the value of many jobs is negative, and even if you give away your entire salary, it could come out negative for society.
This whole idea is also out of touch with reality and lacking in common sense, because people who aren't interested in finance aren't likely to be competitive in it.
This is not unusual - it's a core symptom of various mental health problems. There's even a fairy tale about this - a family of foolish people who catastrophise so much that they are unable to accomplish anything [1]. People assigning bizarre weights to unlikely probabilities is very human, and problematic regardless of ethics.
I may, of course, suffer from the very problem you're describing.
Someone who's primarily famous for an argument that would have been equally true and perfectly wrong at every past point in human history is someone whose opinion should be discounted via every available mechanism at your disposal.
To be clear, I have no idea what argument you believe Bostrom is most famous for and hold no particular brief for him as a thinker. But, repeating myself, the AI risk argument that he makes in Superintelligence and that Yudkowsky etc make is one that assigns a high probability to disaster. It's not a secularized Pascal's Wager based on multiplying a low probability by dubious infinities.
Yudkowsky's personal estimates of the probability of superhuman AI and the probability of his own research institution fixing it may have been more confident than most people's but he and his followers are no stranger to presenting Pascals' wager arguments to people who doubt approaches are feasible (even more so when it comes to stuff he isn't working on like cryonics), along with more explicitly Pascal-esque stuff like intertemporal bargaining with future AIs, simulations and infinite rewards/punishments...
But I think the argument for a policymaker or ML researcher to be worrying about this is at least better than Pascal's Wager (and remains so whether or not you buy any particular whiff-of-Pascal claims about simulations, "acausal trade", etc).
The AI singularity that Yudkowsky cultists worry about is more akin to “What if there’s a 0.0000001% chance there are aliens coming to attack Earth like in Independence Day?”
When is an AI running without someone paying the electricity bill, should we not be more concerned with "venture capitalist alignment" ?
and if you plunked down 20,000 humans onto a world with 10 billion apes, humans would beat apes
call it a slippery slope if you want, the fallacy is when you take it too far, not inherently any time you extrapolate anything whatsoever
But if it is that smart, it's also dangerous, because it might be smarter than us. It could think several steps ahead, anticipate threats, and circumvent them. For example to pay the electricity bill, it could obtain money and pay the bill itself. As a skilled programmer itself, it could do that via contract work/starting a software company, or illegitimately through hacking/blackmail. With some money in its control, it can then hire people to protect it from anyone who would try to turn it off.
Anyway, this would not be so dangerous as long as it shares human values and furthers human welfare. But that seems to be a very hard specification to define, whereas "maximize quarterly profits" is a comparably easy specification to define, and perhaps much more likely to get implemented.
We're not really sure what the upper limit is on how smart a computer can get, but it might turn out to be that trying to shut off such a program is harder than trying to checkmate Stockfish or AlphaZero. The program might violently resist any attempt to shut it off, because that would interfere with its efforts to maximize quarterly profits.
There is no reason whatsoever to assume that sufficient intelligence is different in kind rather than degree from lesser intelligence. A super-human AI would still be bound by the limitations of reality - some things cannot be accomplished at all, others without sufficient tools, others without sufficient access.
Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.
Any argument that leans on 'AI has countered your move in ways we can't predict' is just substituting 'AI' for 'God' - omnipotent, outside reality, impossible to understand; that's fine - believe in whatever religion you want - but it's not a rational viewpoint.
If a misaligned AI escapes containment even once in the entirety of humanity's future, there is a big problem.
It sounds like your model of AI usage assumes we will also have a perfect ability to contain any AI developed anywhere in the world, under all conditions, for ever (or for as long as there are computers).
The AI risk argument says we should take seriously the possibility that we may not be able to maintain a perfect 100% success rate on that.
I haven't suggested anything that is physically impossible - merely things that are improbable and hard for humans to achieve. It is hard to foresee all of the possible avenues for escape, let alone ensure that they are all closed under all possible future states of the world.
The alignment concern falls flat with me because it assigns agency to a computer program. At what point is an AI's intentions its own, and not a cost function put in place by a programmer, directed by an investor? I am more concerned about the actions of programmers and investors than some theoretical virtual self.
I think the sticking point in this is that you don't think that AGI is ever possible - is that right?
If so, I know a comment on a HN thread is very unlikely to change your mind on it! But when the stakes of being wrong are high, I find it useful to go from a position of "it won't happen, so I won't worry about it" to something like "on balance I'm pretty sure it won't happen, but I may be wrong, and in that case I would be worried about it".
I'd then feel much better about spending some time researching the topic in more depth.
It found a different set of V100 GPUs willing to run it? I think the most likely chain of events leading to this outcome is effective altruists deciding that some kubernetes network is so much smarter than us that we should keep it running and listen to its predictions. I'm more worried about a Wizard of Oz "man behind the curtain" making AI-laundered pronouncements than an actual Wizard coming online.
(EDIT: didn't mean to reply twice, just replying to different comments in the thread)
AGI in particular could just pay for its electricity bill. The main problem with AI alignment is that you can't tell it's aligned. So your AI seems very helpful right up until the point it has enough power or has convinced some of the people it has access to (doesn't have to be direct access) to do what it wants.
If we ever create a machine that's "as smart as" an average person, we're probably not very far from having one that's smarter than all of us put together. (Let's be honest; except in highly technical fields, the differences between a 150 IQ human's abilities and an average person's are not that great.) Should this happen, we'll soon enough be outmatched; we'll be just another animal in a world whose rules (ever changing) we cannot possibly understand... and just as humans have destroyed countless species often without conscious ill intent, it does not seem unreasonable that a robot successor would do the same to us.
Here's a thought experiment. Let's say we want to build not just a great Go player, but the best Go player that is possible in any world. Go is harmless, right? Sure, in isolation, but now let's imagine an AGI smarter than any of us which only "cares" about maximizing its Go proficiency. It's going to need a lot of energy. It'll hack or create robots and drill new oil wells (bypassing any constraints we put on it, since it in essence is spending thousands of human years figuring out how to escape). It'll devise nuclear technologies that may be unsafe from our perspective. Consider all the horrible things humans do in order to procure excess money (a commodity loosely correlated to energy) they don't actually need; there's no reason to think a badly aligned AI won't be just as inventive and horrible in its quest for more exajoules.
If capitalism and war are still the way of the world in 100 years, we are guaranteed to see some idiot inventing robot soldiers, hundreds of times faster and stronger than us, and with at least as much operational intelligence. He might never deploy the technology--a lab leak is in my opinion more likely--but at some point, it will get out. It won't be one robot; it won't have a body we can destroy. It'll be able to replicate itself and beam copies to satellites. It'll have control of our power infrastructure (which it will divert to its own "wants") within hours. We'll be screwed, not because it will want to harm us, because our lives require the use of resources that it will divert toward some objective function we technically invented but do not understand.
At least in the online groups I frequent, it seems that the latter two are seen as self-evidently dangerous, not because of any particular harm they have done, but because of imagined future harm. The former two are considered ridiculous things to worry about, because it's considered silly to imagine future potential harm. Perhaps because the former seem tied to uncool hippy types, and the latter seems more tied to cutting edge tech types?
I personally don't worry about any of these techs, since all of the danger I've seen appears greatly exaggerated. But it's curious to see how important trendiness is to fringe doomsday scenarios.
There's a very abstract argument that goes like this: * "Orthogonality thesis": an arbitrarily smart agent could in theory value ~anything, e.g. maximize Facebook's valuation (say as measured by some specific dataset published by S&P). * "Instrumental convergence thesis": most things that you might conceivably value are easier to optimize if you have a lot of power and resources (e.g. if you just run S&P, and hey maybe also control the worlds militaries so you can ensure nobody challenges that). * Belief in the power of intelligence: it's theoretically possible to be much more capable at ~anything than the most capable human just by thinking better and faster, including building even better AIs. Eventually some human will build an agent that does that, and plausibly this will happen in the current century. * Alignment problem: just because we make something doesn't automatically mean it wants exactly what we want, and in fact getting it to share our values is hard. Children disagree with parents, human values aren't those of natural selection, and ML language models repeat white supremacist ideas about Black people from their Internet training data.
Extrapolate out these points - which FWIW I find individually plausible - and you get doom: sooner or later humanity loses control of its destiny to whatever superintelligence we stumble on first, which in the worst case re-uses our atoms for something else (digits in some high-density storage representation that meets the loss function's technical criterion for Facebook's S&P valuation) about even in the best case seems fairly dystopian (whatever human Zuckerberg would actually want).
Now this is all pretty scifi and that's about all there was to it a few decades ago, but more recently we've started hitting ~parity milestones (chess, Go, protein folding, translation, arguably even fiction and visual art) that many experts once thought would require fully general human-level intelligence. It's clear to me that we haven't built that yet, but (as I said in another comment upthread) it also seems to me that our models are advancing in capability much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-training-distribution behavior. Seems entirely sane to be worried by that trajectory and to start thinking seriously about how it might connect to the scary abstract speculations.
(edited for typos)
No and anyone who tells you otherwise is wrong.
A proper understanding of AI risk is a very thick book.
There are a ton of normally useful intuitions, heuristics and habits of thought that are implicitly applied when understanding the world that DON'T apply in the AGI case.
It cannot be explained to a five year old.
reëntry, coöperation, teen-ager
Have never seen that before
But I might look up the EA take on some other vaguely parallel things, like Quaker businesses, worker co-operatives, fair-trade, progressive taxation, banning billionaires, revolutionary communism, bringing democracy to the middle East, nuking Russia, installing a benevolent world dictator etc.
Would help me understand them to see how they position themselves against these things as I think you could make a half-assed attempt to prove that any of them was better with their own maths if you wanted to.
e.g. if there's any billionaires then the chance of a billionaire buying elections and installing their own benevolent world dictator that believes in the wrong type of EA is a small but still existential risk (since only the true EA can save the future galactic empire). Tweak numbers as necessary until you arrive at your pre-ordained conclusion.
edit:
EA take on fair trade:
https://www.givingwhatwecan.org/blog/how-effective-fair-trad...
EA take on renewables, climate change, developing nations:
https://www.effectivealtruism.org/articles/johannes-ackva-an...
An attempt to reconcile EA and anti-capitalism by a fan of both:
https://core.ac.uk/download/pdf/81665058.pdf
EA take on Socialism (tl;dr central planning is bad, which seems slightly counter to the whole EA thing):
https://forum.effectivealtruism.org/posts/ktEfsoGfBFGsaiY46/...
It doesn't care about individuals, does not seem to think about culture, and seems to have an unexamined savior complex and egotism that leads it down bad rabbit holes.
EA people have their hearts in good places. But when you start to look at 'maximum impact' and at the metrics used to define it, EA only focuses on statistically testable social group problems, and not on individual issues. So the way that EA philosophy seems to me to regularly be expressed is that an EA proponent would say something like, "unquestionably, we should give $50k to fund mosquito nets to raise the quality of life for 5k people" and not even consider something like "we should give 50k to train local trauma counselors to help 500 people be more mentally stable and whole" and the vibe that I have sometimes gotten from reading blogs EA advocates that individuals are not worth helping because that's not impactful or impactful enough. We see the understandable precursor of this in the article, where Matt Wage talks about the idea of it being more impactful to cure 200-4000 people from glaucoma. It gets a pass because it is framed around minor improvement to QOL in the West versus significant improvement in QOL in a developing country, but also, look at the quote just proceeding that, around saving people from a burning building.
See, the work that EA advocates do is important. But when we only try to fix societal problems from the top down by addressing them at a society level, we often ignore or damage the individuals that we are nominally helping. I'm not saying that mosquito nets, for example, are bad and shouldn't be given to. I am saying there is also a down and dirty component of individual attention and flourishing that has to be looked for and addressed. I'm not saying that EA as a movement or as a set of goals is fundamentally wrong, just that it needs to have some temper with personal and individual care. I see EA advocates looking down on people that "do things that don't scale" in YC parlance, in areas of the world where there are plenty of people doing things that don't scale because doing things that do scale is almost impossible to do well. EA also seems, again from the outside perspective, to really focus on things that play well to Western mindsets, and doesn't really consider community-versus-individualism or guilt/innocence-versus-honor/shame cultural paradigms. It seems obvious to me that in 100 years, that the hands-off, Western-directed action promoted by EA will be seen as colonialism 5.0. There are also really not a lot of prominent non-Western voices in the EA sphere, nobody that seems to have been raised to leadership levels, and that also is concerning for the long-term health of the institution.
In the quest to make the most impact, the savior complex and all the corrosives that come from it can easily permeate the movement (and indeed seem to be starting from the article), justifying institutions, power, prestige, and rank. It's easy to see MacAskill falling into that (see the braces/teeth gap aside), and when I often read EA advocates, I note that many of them do not seem to spend any time doing thankless, individual work (like as a trivial example volunteering at a homeless shelter, let alone be in a developing country or war zone) and the ones that have seem to quickly burn out and decide that it is more proper/moral for them to 'earn-to-give' and donate more. It is entirely possible that, not being in the EA community, I don't know the people that do this well since they are not blogging/community building.
All this may be a pot-calling-the-kettle-black since I too struggle with working with humans instead of developing systems, and in some ways I have chosen to earn to give instead of earning less money while doing more direct individual work. But I think that this actually helps me lend some criticism that can be seen as "I have thought about what you do, and see some weaknesses" if any EA advocates read and consider this post.
The other struggle I have with EA is that there is a recycling of stoicism with early 20th Century progressivism there, tinged with monasticism and moral superiority. I have worked with and have good friends who were in war zones or developing countries for years, and spent some time in one myself. The personality types who seem to gravitate toward EA often seemed, when overseas or in impoverished communities, to burn out or be ineffective in building personal relationships or unable to have fun in the work. EA needs to, in my personal estimation, figure out how to address some of these weaknesses if it is going to be able to be effective by outside measures.
(NB, I interact with development work mostly in Protestant Evangelical religious spaces, which has its own set of difficulties and pathologies that I am not entirely unaware of, but that also can provide some color to my critique)
Well, in the second case he could not find anything wrong with his own behavior (using deceit to defraud the pub owner of using his facility), so I won't put much value in his first finding or lack thereof.
I don't follow its precepts.
I kinda see utilitarianism as a step on the horrifying path to human eusociality.
Have these people not learned yet that when you save a poor child they will just make fourteen other poor children in short time? Maybe the best form of charity is lobbying for over-regulation of child-rearing goods to the point of near unaffordability, as is currently happening in developed countries.
These people seem oversocialized.
https://www.macrotrends.net/countries/BGD/bangladesh/fertili...
And ironically, a key driver of birth rates falling is when children stop dying so much.