EA cult members do not take this very real, measurable, non-speculative danger seriously. Consequently, I don’t think they should be trusted or consulted on any subject of any importance.
EA cult members do not take this very real, measurable, non-speculative danger seriously. Consequently, I don’t think they should be trusted or consulted on any subject of any importance.
https://news.ycombinator.com/item?id=49885784
But societal collapse would save us, and international coordination is theoretically possible, so I put it only around 90%.
1. Human values are a result of our extraordinarily complex shared cultural and evolutionary history, and accordingly are not shared by any AI, or even possible for us to formally define.
2. We do not know how to impose human values on an AI (note that this isn't the same as teaching an AI to model human values; the agents in the various hacking incidents knew their actions conflicted with human values, but their own values were only to maximize their predicted reward scores).
3. Intelligence is orthogonal to values. Increasing intelligence does not naturally cause values to converge on human values.
4. Sufficiently superior intelligence allows you to impose your values on beings with inferior intelligence. This implies recursive self-improvement is a logical sub-goal of all unbounded goals.
5. Human intelligence is not close to physical limits. This implies recursive self-improvement is possible.
6. Somebody will give an AI an unbounded goal. This is already the standard (maximize reward score).
I haven't seen any convincing counterarguments to any of these. Most people claiming AI development is safe don't even address them.
It's butlerian jihad or bust, I'm afraid.
That's not exactly true.
"Climate change matters so much, to so many, not just because of the suffering and injustice it’s already causing, but also because it’s one of the few issues that has obvious potential to affect our world over many future generations. We think safeguarding future generations is a key moral priority, and should be a crucial consideration in prioritising problems on which to work.
...
...climate change will be hugely destructive. We’ll see floods, famines, fires, and droughts — and the world’s poorest people will be affected the most.
...
...climate change’s impacts will still be significant – it could destabilise society, destroy ecosystems, put millions into poverty, and worsen other existential threats such as engineered pandemics, risks from AI, or nuclear war. If you want to make climate change the focus of your career, we include some thoughts below on the most effective ways to help tackle it.
...people are right to be angry that too little is being done.
...
Working on this issue seems to be among the best ways of improving the long-term future we know of..."
https://80000hours.org/problem-profiles/climate-change/
A big part of the reason EA doesn't focus more on climate change as a "highest priority area" is simply that many people are already focused on it, and it is therefore not an especially "neglected" area.
And then we raced ahead straight into materializing the x-risk everyone thought is still a few decades away, speedrunning through all the mistakes LW folks itemized and worried about over the past two decades.