Be a Visiting Fellow at the Singularity Institute
lesswrong.com
lesswrong.com
Forgive me, but this seems a little on the "self-important quackery" side. There are any number of threats to our species far more urgent and reality-based than xenocide by a weakly godlike post-singularity AI. I suppose it can't be completely ruled out any more than alien invasion or the Nemesis hypothesis (though I find it pretty far fetched given the state of AGI). But "the world's most important task"? C'mon.
We work on preventing those too. Some of the risks we work to prevent:
- Deliberate misuse of nanotechnology
- Nuclear holocaust
- We’re living in a simulation and it gets shut down
- Genetically engineered biological agent
- Accidental misuse of nanotechnology (“gray goo”)
- Physics disasters
- Misguided world government or another static social equilibrium stops technological progress
- “Dysgenic” pressures
- Take-over by a transcending upload
- Repressive totalitarian global regime
- Our potential or even our core values are eroded by evolutionary development
For more information, see http://www.nickbostrom.com/existential/risks.html.
That being said, I think the way you state your mission appears a little immature to me and many other people, and I think that very seriously detracts from your credibility. This is very important, because with more credibility you could accomplish many more things that make the world a better place. One of the most important things about scientific research is having a concrete, well defined way to attack a problem you're setting out to solve. People are working on quantum mechanics because there are lots of concrete things they can do and test, and people aren't working on antigravity because they don't know where to begin. Your research is from the "antigravity" category of problems, i.e. problems where we don't really have an attack.
This doesn't mean working on these problems isn't useful, or that you should stop, or that you should spend your time doing something else. It does mean that your research is highly philosophical - it is very interesting intellectually, it has potential to attract great young minds to work on interesting things, and it will likely produce results that will get people thinking in new directions. This is always a good thing. But it also means that you are extremely unlikely to find practical solutions to any of these problems, and it seems that everyone is recognizing this other than you. When you present yourself as an organization that is working on practical problems, instead of intellectually interesting philosophical problems, it becomes very difficult for many intelligent people to take you seriously, and I think it significantly detracts from your goal.
Once again, I think what you're doing is great, I wish more people were doing it, and I sincerely wish you luck and success. I am merely trying to show you how many intelligent people that would otherwise be interested in what you're doing view your organization. Please treat this comment as a constructive argument for changing your marketing. At the very least, think seriously about slightly adjusting the wording of your mission statement. It might get more of the right people interested, which will likely accelerate your work, and make all of us better off for it.
Also, I'm waiting for SLAC to burn through their barrier and burrow through Palo Alto.
The probability of such a global failure occurring within the next 100 years strikes me as near 1.0, and thus possibly more pressing than any of the events listed above.
http://www.nickbostrom.com/existential/risks.html
In these terms this could be a 'crunch'.
So suppose we are worried that a die-off in industrial nations will actually make it impossible. Isn't this risk (that there is a major catastrophic disaster for industrialized nations and this disaster prevents a singularity from ever taking place) much smaller than the risk that the singularity goes badly when it happens? If so, then this supports the spirit of my original comment: worrying about the singularity going well is more pressing than worrying about major shocks to civilization which do not completely wipe out humanity.
Isn't the risk that a large industrial collapse happens and it prevents a singularity from ever taking place much smaller than the risk that the singularity goes badly? If so, then this agrees with the spirit of my comment: worrying about the singularity going well is much more pressing than worrying about catastrophes that do not immediately wipe out humanity.
Is this belief based on your familiarity with and assessment of the best(1) arguments(2) for and against Singularity/AI risk? If not, then what? Do you think the idea is absurd enough to confidently dismiss without engaging, and if so, why?
(1) http://lesswrong.com/lw/lw/reversed_stupidity_is_not_intelli...
(2) e.g. http://singinst.org/AIRisk.pdf, http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_...
Debating the likelihood of recursively self-improving AI is tedious, usually pointless, and often goes the way of political and religious debate, which could hint at its inclusion in the participants' self definition [1], and I don't have the time or energy to get involved right now.
I apologize if I came off as dismissive -- I was frusterated over my initial point. If it matters to you, I certainly haven't completely written off recursively self-improving AI. I just find it extremely unlikely in the near-to-mid future, and think there are other more pressing issues that have the distinction of currently existing.
Is shaping the singularity the same as averting human extinction? If the singularity itself poses existential risks (this seems obvious,) and if extinction is inevitable without a singularity (this seems likely,) then we need a singularity that is 'safe and soon'. (though the initial wording does seem to gloss over the need to not go extinct in modern ways.)
Put another way: We can do the most work with the most powerful tools. There is a great need to use very powerful tools correctly. If we can even maybe influence the work done by highly powerful mid-singularity tools, we can accomplish far more 'correct work' than we could just using our modern tools. (Again, this does require that we survive long enough for the singularity to happen, and that you don't have a strong discount for future vs. present utility.)
Personally, I think creating a super-tool which can only be used correctly is the best approach.
The answer is trivially "no". It's easy to imagine a future in which we continue to exist and the Singularity never arrives. Perhaps the rate of progress will continue to increase at a decreasing rate. Perhaps it will go linear, or even sigmoid. Perhaps we will achieve a Kardashev Type I civilization without ever going properly singular. It's not a given and perhaps not even likely that singularity is synonymous with the survival of our species.
We have the existence proof for general intelligence embodied in about ~3 lbs of protein. No more, no less. Being certain of the inevitability of the Singularity is closer to religious than skeptical thinking.
Also, for certain definitions of inevitable, extinction is inevitable without a Singularity. A Type 1 Civilization could at absolute best colonize a few surrounding solar systems, but eventually the stars will burn out. Though a universe where we hit Type 1 but never Type II would be rather interesting.
Though I suspect you'll have to lure Clippy here with the promise of persuading angel investors into funding nanotechnological paperclip foundries.
I'm not saying "If humans continue to exist, we will certainly do the singularity thing." I'm saying "We are likely going to go extinct unless things change in a really really big way." And of course those big changes also have to not kill us.
Can you explain this point further? If we can manage to, say, genetically engineer ourselves into healthy immortals that are otherwise pretty much as we are today, invent FTL drives to spread ourselves across the universe, and discover a way to reverse entropy to avoid the big crunch—and after doing all that we still haven't turned ourselves into para-omniscient universe-scale entities—then do we really need any sort of technological singularity, or can we just decide to hold off on it indefinitely?
There probably are ways we could steer into a stable human populated reasonably pleasant universe, but we're in a car filled with guns liquor and dynamite on a winding mountain road in the dark.
You're inverting the causation. They're working on it because they think it's important, not vice-versa.
It may be reasonable to think some other task is more important than this one, or for there is no task which you can identify as the most important, but it is in no way arrogant from someone to conclude that there is a most important task, or that some particular task is it, even by a large margin.
I mean, is the counterargument that there is no most important task? or that we can't know it? Or that it's arrogant to think we have it right?
http://lesswrong.com/lw/qk/that_alien_message/
http://lesswrong.com/lw/rk/optimization_and_the_singularity/
Deleted comment
A) An infinitely intelligent machine is not capable of breaking the laws of physics so they don't get god like powers.
B) Intelligence and creativity are always bound by insanity. (You can always over fit any data set.)
C) Sensors limit the acquisition of new information. (You can deduce the existence of quarks from looking at a web cam pointed at a brick.)
D) While processing power has grown exponentially the cost of fabricating that technology has continued to increase faster than world GDP.
B) This is nonsensical ...
C) You can deduce a ludicrous amount of information from relatively little sensor input. Humans don't extract a fraction of a percent of the maximum (in the information theoretic sense) information from our environment. See http://lesswrong.com/lw/qk/that_alien_message/
D) This is trivially, factually incorrect. A 3GHz P4 is cheaper today (in absolute or relative terms) than it was when it was brand new. A new Core 2 Duo will give you better performance/dollar than a P4 when it was new (which was better than a P3, which was better than a PII, etc).
The parent mentioned the cost of "fabricating that technology", not of processors themselves; this is vague, but AFAIK, fabs are getting more expensive quickly.
"The increase in cost and complexity is encapsulated in Rock's Law, which dictates that fab construction costs double every four years. That law is itself tightly connected to the better-known Moore's Law, which dictates that the number of transistors on a chip doubles every two years." (http://news.cnet.com/A-fab-construction-job/2100-1001_3-9810...)
"By 2007, the price of building a fab is expected to reach $6 billion."
$6 billion * 2 ^ 10 = 6 trillion in 20 years.
B) Not really it's a well known problem with AI. Over train a learning system and you get worse behavior.
C) Nope. Information theory puts strict limits on how much you can zoom in with a web cam. If an AI want's to earn about sub atomic physics it needs to look at the output of a particle accelerator. Can it make a better theory with existing data? Well see B.
D) http://en.wikipedia.org/wiki/Rock%27s_law The semiconductor industry has always been extremely capital-intensive, with very low unit manufacturing costs. Thus, the ultimate limits to growth of the industry will constrain the maximum amount of capital that can be invested in new products; at some point, Rock's Law will collide with Moore's Law.
Note that "Singularity" doesn't always refer to exponential growth (http://yudkowsky.net/singularity/schools); SIAI in particular focuses on the "intelligence explosion" definition.
I went there with low expectations of actually accomplishing anything. I'm not yet at the point where I can write the sort of hardcore sciencey analytic philosophy papers that are the main output of the Singularity Institute, but I think with 6 months of learning I will be able to meaningfully contribute to that goal.
I tried to share my lessons of entrepreneurship with the housemates that wanted to learn. I sat for an afternoon with a Brazilian entrepreneur living at the SIAI house and we went through all of the logistical details of opening a Delaware corporation as a foreign national. I did a short workshop on SEO. I went to see Thom Yorke twice in Oakland, saw Orbital in San Francisco, and Blue Scholars in Berkeley.
I graduated from school with an engineering in December and because of the success of my kratom business, I didn't need to get a real job, so I had a lot of time to surf the internet. I started reading Less Wrong while trying to actively comment as much as possible. In 4 months of posting on Less Wrong, I learned more than I did in two years of college. In 3 weeks of living at the SIAI house, I learned much more rapidly than I did while just posting on Less Wrong.
So, to answer your question, what I mostly did was sit around and talk with all of the wonderful people that live at the SIAI house or occasionally pass through or hang out. My favorite types of conversations are those where you push the barriers of the kinds of thoughts that humans are capable of having, the kinds of conversations that often stop making sense because there are too many different infinities or recursive loops. At the SIAI house, I could spend hours a day talking about those kind of things. It sometimes felt like I was living at the center of the universe.
Also, the food is good. Lots of delicious healthy things from Trader Joe's and Costco.
For anyone that didn't get the reference, my primary business at this point is selling legal drugs (sometimes known as "herbal medicine" depending on the conversational context). My website is http://getkratom.com and in the early days of Hacker News (Startup News?) I started an enormous flame war while trying to start a discussion about my business. I have since tried to stop mentioning it here, though I certainly don't mind talking about it to anyone that may be interested.
Usually the way of gauging process is that people come up with specific goals and then either meet those goals or don't meet them. I didn't have a whole lot of goals, which means that whatever it is that I did accomplish was higher than my expectations.
I would recommend that other people go in with high expectations of accomplishing things.
How was it like a startup?