1,339 karma · joined June 12, 2010
Well we work on global warming even though the worst effects are likely decades off. I think one computer science professor put it something like this: "If experts told us aliens were likely arriving on Earth in 50 or 100 years, it'd make sense to start preparing now." You don't want to wait until it's too late to do whatever needs to be done.
Berkeley AI prof Stuart Russell's response goes something like: Let's say that in the same way Silicon Valley companies are pouring money in to advancing AI, the nations of the world were pouring money in to sending people to Mars. But the world's nations aren't spending any money on critical questions like what people are going to eat & breathe once they get there.
Or if you look at global warming, it would have been nice if people realized it was going to be a problem and started working on it much earlier than we did.
Let's say we program an AI and get one little detail wrong and things go to hell as a result. We can call that "human error" or "AI error" but either way it's a reason for caution.
I am actually somewhat concerned by this OpenAI project, and here's why. Let's say there's going to be some kind of "first mover advantage" where the first nation to build an AI that's sufficiently smart has the possibility to neuter the attempts being made by other nations. If there's a first mover advantage, we don't want a close arms race, because then each team will be incentivized to cut corners in order to be the first mover. Let's say international tensions happen to be high around this time and nations race to put anything in to production in order to be the first.
The issue with something like OpenAI is that increasing the common stock of public AI knowledge leaves arms race participants closer to one another, which means a more heated race.
And if there's no first mover advantage, that's basically the scenario where AI was never going to be an issue to begin with. So it makes sense to focus on preparing for the more dangerous possibility that there is a first mover advantage.
>...imagine a hypothetical computer security expert named Bruce. You tell Bruce that he and his team have just 3 years to modify the latest version of Microsoft Windows so that it can’t be hacked in any way, even by the smartest hackers on Earth. If he fails, Earth will be destroyed because reasons.
>Bruce just stares at you and says, “Well, that’s impossible, so I guess we’re all fucked.”
>The problem, Bruce explains, is that Microsoft Windows was never designed to be anything remotely like “unhackable.” It was designed to be easily useable, and compatible with lots of software, and flexible, and affordable, and just barely secure enough to be marketable, and you can’t just slap on a special Unhackability Module at the last minute.
>To get a system that even has a chance at being robustly unhackable, Bruce explains, you’ve got to design an entirely different hardware + software system that was designed from the ground up to be unhackable. And that system must be designed in an entirely different way than Microsoft Windows is, and no team in the world could do everything that is required for that in a mere 3 years. So, we’re fucked.
>But! By a stroke of luck, Bruce learns that some teams outside Microsoft have been working on a theoretically unhackable hardware + software system for the past several decades (high reliability is hard) — people like Greg Morrisett (SAFE) and Gerwin Klein (seL4). Bruce says he might be able to take their work and add the features you need, while preserving the strong security guarantees of the original highly secure system. Bruce sets Microsoft Windows aside and gets to work on trying to make this other system satisfy the mysterious reasons while remaining unhackable. He and his team succeed just in time to save the day.
>This is an oversimplified and comically romantic way to illustrate what MIRI is trying to do in the area of long-term AI safety...
http://lukemuehlhauser.com/a-reply-to-wait-but-why-on-machin...
Nuclear weapons come to mind. Would we prefer that the knowledge of how to make them be more widespread?
* Two guys with pistols in a crowded bar: Attacker is favored.
* Trench warfare during WWI: Defender is favored.
* Nukes: Attacker is favored, although with the invention of nuclear submarines that could lurk under the ocean and offer credible retaliation even in the event of a devastating strike, the attacker became favored less.
In general equilibria where the defender is favored seem better. Venice became one of the wealthier cities in the medieval world because it was situated in a lagoon that gave it a strong defensive advantage. The Chinese philosopher Mozi was one of the first consequentialist philosophers; during the Warring States period his followers went around advancing the state of the art in defensive siege warfare tactics: http://www.tor.com/2015/02/09/let-me-tell-you-a-little-bit-a...
Notably, I'm told that computer security current favors attackers in most areas: http://lesswrong.com/lw/dq9/work_on_security_instead_of_frie... (BTW the author of this post is a potential Satoshi Nakamoto candidate and knows his stuff.)
In equilibria where the attacker is favored, the best solution is to have some kind of trusted central power with a monopoly on force that keeps the peace between everyone. That's what a modern state looks like. Even prisoners form their own semiformal governing structures, with designated ruling authorities, to deal with the fact that prison violence favors the attacker: http://www.econtalk.org/archives/2015/03/david_skarbek_o.htm...
Thought experiment: Let's say someone invents a personal force field that grants immunity to fists and bullets. In this world you can imagine that the need for a police force, and therefore the central state authority that manages the use of this police force, lessens considerably. The enforcement powers available to a central government also lessen considerably.
This is somewhat similar to the situation with online communities. We don't have a central structure governing discussions on the web because online communities favor the defender. It's relatively easy to ban users based on their IP address or simply lock new users out of your forum entirely and therebye keep the troll mobs out. Hence the internet gives us something like Scott Alexander's idea of "Archipelago" where people get to be a part of the community they want (and deserve): http://slatestarcodex.com/2014/06/07/archipelago-and-atomic-... Note the work done by the word "archipelago" which implies islands that are easy to defend and hard to attack (like Venice).
Let's assume that superintelligent AI, when weaponized, shoves us into an entirely new and unexplored conflict equilibrium.
If the new equilibrium favors defense we'd like to give everyone AIs so they can create their own atomic communities. If only a few people have AIs, they might be able to monopolize AI tech and prevent anyone else from getting one, though the info could leak eventually.
If the new equilibrium favors offense we'd like to keep AIs in the hands of a small set of trusted, well-designed institutions--the same way we treat nuclear weapons. It could be that at the highest level of technological development, physics overwhelmingly favors offense. If everyone has AIs there's the possibility of destructive anarchy. In this world, research on the design of trustworthy, robust, inclusive institutions (to manage a monopoly on AI-created power) could be seen as AI safety research.
The great filter http://waitbutwhy.com/2014/05/fermi-paradox.html weakly suggests that the new equilibrium favors offense. If the new equilibrium favors defense, even the "worst case scenario" autocratic regimes would have had plenty of time to colonize the galaxy by now. If the new equilibrium favors offense, it's entirely possible that civs reaching the superintelligent AI stage always destroy themselves in destructive anarchy and go no further. But the great filter is a very complicated topic and this line of reasoning has caveats, e.g. see http://lesswrong.com/lw/m2x/resolving_the_fermi_paradox_new_...
Anyway this entire comment is basically spitballing... point is that if $1B+ is going to be spent on this project, I would like to see at least a fraction of this capital go towards hammering issues like these out. (It'd be cool to set up an institute devoted to studying the great filter for instance.) As Enrico Fermi said:
"History of science and technology has consistently taught us that scientific advances in basic understanding have sooner or later led to technical and industrial applications that have revolutionized our way of life... What is less certain, and what we all fervently hope, is that man will soon grow sufficiently adult to make good use of the powers that he acquires over nature."
And I'm slightly worried that by virtue of choosing the name OpenAI, the team has committed themselves to a particular path without fully thinking it through.
AI luminary Stuart Russell also takes on this analogy in this presentation: https://www.cs.berkeley.edu/~russell/talks/russell-ijcai15-f...
>OK, let’s continue [the overpopulation on Mars] analogy:
>Major governments and corporations are spending billions of dollars to move all of humanity to Mars [analogous to the billions that are being spent on AI]
>They haven’t thought about what we will eat and breathe when the plan succeeds
>If only we had worried about global warming in the late 19th C.
If regulations do turn out to be the right path, I'd suggest that people within the AI field form their own informal regulatory body first, fortifying it against institutional failure modes like corruption by lobbyists etc. Then get the government to grant them legal authority. Hopefully that would go a ways towards addressing the issues you describe.
Interesting that you mention that. Charity evaluator Givewell has evaluated hundreds of charities based on whether they actually deliver results for your money. One of their top rated organizations is GiveDirectly, which does almost exactly what you describe. Here's their evaluation page for it:
http://www.givewell.org/international/top-charities/give-dir...
Givewell is a bit bearish on disaster giving, by the way. See #6 in this list:
http://blog.givewell.org/2013/11/12/6-tips-on-disaster-relie...
If you're interested in this sort of thing, you might look in to the Effective Altruism movement, which I consider myself a part of:
The reason we haven't started our crowdfunding campaign yet is because our following isn't big enough yet. Soylent hit its target within the first 24 hours because the DIY recipe already had a huge following, and the general wisdom for kickstarters seems to be that you need at least 30-40% of your funding nailed down before you hit the "go" button. So in addition to feedback on the product itself, we are curious if HN has any thoughts for things like slogans and other product messaging, and customer verticals that we might be able to market the product to.
There's also the possibility that you will find something that really motivates you, but it ends up falling pretty wide from what you originally wanted to learn--e.g. maybe you find that you really love programming, but only if it's in TI-BASIC on your graphing calculator.
Are you sure? Who's to say the super-smart couldn't help us with other more mundane problems like understanding each other better, being more productive, thinking more rationally, etc.? I've had useful insights on these three topics that seemed very clever to me and I know of friends who have also had such useful, clever insights.
At the top of any HN comment thread, you're likely to see some clever, apt insight on a topic unrelated to science & math.
Regarding your examples, Marilyn vos Savant's IQ seems pretty sketchy: https://en.wikipedia.org/wiki/Marilyn_vos_Savant#Rise_to_fam... I don't know what evidence you have that Scott Adams is supersmart; he's a Mensa member, but I suspect many HN users could become Mensa members if they wanted to (an acquaintance reports that the intellectual caliber of a Mensa meetup is around that of a science fiction convention; the top 2% of the population is simply not that high of a bar).
I do agree that after you reach a certain IQ, determination, habits, ambition, rationality, etc. become larger determining factors in your level of success. But if you are smart enough you can optimize those manually for yourself ;)
What I'm about to say is probably obvious to most HN readers, but I'll say it anyway in case the author is reading the comments: right now the large majority of commercial software is closed-source, and teaching everyone to code is probably not going to change that. Even if it did somehow change, and there were a bunch of eyes looking over all the software that companies wrote, it would probably make it an expensive PR blunder to not implement privacy best practices, etc... you wouldn't want your software's failure to implement proper protections to be a top story on HN.
http://lesswrong.com/lw/h3q/a_rational_altruist_punch_in_the...
I wish he wouldn't try to figure out what hidden motive prevents people from following his strategy before it's even been established as the best strategy.
Looks like it ended in late 2007.
That works great when you have a friend to bug. And nowadays, you can Google for solutions, so this kind of "I'll learn it when I need it" strategy works reasonably well.
But what doesn't work is when you have no friend to bug, and you don't have enough knowledge even to know what Google keywords you should be using. That's the point at which you really want to spend time acquiring fundamental concepts without regard for how useful they'll be in the near term.
Let's say someone was trying to learn physics, and they didn't yet know algebra. Which strategy do you think would work better: making a dedicated effort to learn algebra first, or diving in to physics and only learning algebra as they needed it?