What If the Singularity Does Not Happen? (2007) [video]

> the FDA and its counterparts are not going to allow applying those technologies the very moment they are invented.

And probably for very good reasons.

"the field of science progresses one funeral at a time"

Those are good points.

In addition many things require experiments or building large scale factories/businesses and those take time.

I don't see this. I would expect the Singularity to be mostly powered by hardware, software, robotics, and manufacturing, all of which are lightly regulated.

obtaining regulatory approval for innovations in various spheres (medicine, aviation) takes forever

If regulatory approval is the real bottleneck then there would be a rapidly-growing backlog of technologies that have been invented but not approved. Do we see that?

people wanting to use their present purchases for some amount of time before shelling out again for the next generation of tech

People are keeping iPhones for 4-5 years instead of 2 years but Apple still has enough R&D budget to release a new model every year.

> Even if we invent nanobots or whatever so quickly, the FDA and its counterparts are not going to allow applying those technologies the very moment they are invented.

I've often wondered if countries (such as maybe China or India) where medical devices/pharmaceuticals/etc are more lightly regulated may some day end up pulling ahead of countries like the US with stricter regulations.

Of course, stricter regulations exist for a reason – to reduce incidence of death or injury from insufficiently tested treatments. But, there is always a risk-reward tradeoff–being willing to tolerate a greater incidence of death/injury due to new treatments may increase the progress of medical science–and some societies may collectively decide to locate that tradeoff at a different position than what the US (or even other Western countries) does.

There's a more basic problem - the Singularity assumes automated self-improvement is possible.

The reality is that most codebases are brittle and bug-ridden. Poor reliability is one the defining characteristics of software systems.

The most reliable systems require formal methods, which require a nailed-down spec. There is no nailed-down spec for an open-ended self-improving system, and it's debatable if a spec is even possible.

So the only practical systems will be heuristic and inherently unreliable in at least some domains, with a tendency to fail spectacularly - much like humans.

This is a basic limitation of technology. You can have mechanical perfection in a limited domain (although with software you often you get a poor imitation of that) or you can trade off flexibility for reliability.

You can't have both.

The singularity isn’t happening because the concept of exponential technological growth is, as a general matter, wrong. In any given area, technology improves rapidly, then plateaus. (From 1903-1969, we went from the Wright Flyer to a moon landing. In a similar period after that, our technological capability in that area has barely advanced.) It’s not like there is incredible technology that’s just sitting around waiting for regulatory approval.

Vinge’s definition of the singularity is the creation of human-level AI. After we have superhuman AI governments will be irrelevant. What ever the AI wants it gets.

You can also read Vinge's essay from 1993:

https://edoras.sdsu.edu/~vinge/misc/singularity.html

Really? How much progress was made on thinking machines in the 19th century? How much between 1900 and 1950? 1950 and 2000? 2000 and 2010? 2010 and now?

I think each of those timespans is very roughly equivalent. In the past 10 years voice recognition has gone from useless to ubiquitous. Neural image labeling and segmentation likewise. Physical robots have gone from taking a few faltering steps to doing parkour. None of that counts as "AI" any more because it works and we know how, but the rate at which new capabilities are moving from the realm of "AI" to the realm of "normal engineering" is increasing exponentially.

I agree that technology isn't progressing "faster and faster," but GDP doesn't measure technological progress. That is absurd. It doesn't even measure economic growth very well, and even if it did that wouldn't mean technological progress would track it. Technological progress can happen even during a recession, and often has.

GDP measures the value of monetized final goods in an economy. Linux and Wikipedia have thus reduced the GDP, because they displaced a huge chunk of existing markets that used to be monetized.

GDP is useful for monetary policy conducted at central banks, primarily.

Great episode of Peter Thiel and Eric Weinstein discussing this slowing of technological progress: https://www.youtube.com/watch?v=nM9f0W2KD5s

The question might be impossible to answer until after the fact, but it's not ill-posed. Machines either will, or will not, kill all humans.

Possibly Vinge is well aware that a superior intelligence would be incomprehensible, but he just hasn't figured out how to write a compelling story about that.

> Questions like "Will the machines allow humans to live?" are ill-posed.

I don't think so. Even if the machine doesn't care about you (it may not even have a notion of what is a human), you are still made of atoms it could use for something else.

No, AI will desperately want sex, money, and lebensraum, just like human dictators. It won't think of either humanity or self-preservation as completely irrelevant and not interesting.

Singularity talk is just projection and self-idealization. "AI will be like me, but perfect, and infinitely more capable of doing what I would do if I were omnipotent, which it will decide to do faster than I could ever make a decision."

I think part of the problem with the question is that intelligence and motivations/drives are orthogonal. Intelligence is just a means to an end, it says nothing about the end to which it is a means. To say that a being is vastly more intelligent than human beings (super-intelligent) says nothing about what motivations it may have. Our own motivations are mostly given to us by our genetic and cultural inheritance, rather than something we've constructed for ourselves by our own intelligence.

I think it likely, that any super-intelligent AI created by human beings, will be given motivations which somehow reflect the motivations and values of its creators. Not necessarily exactly the same – a silicon-based life-form has no need for food or sex as carbon-based life-forms do – but, I think a super-intelligent AI created by humans will likely be given motivations to seek to favour the interests of its creators, whether "its creators" are construed broadly (all of humanity as a whole) or narrowly (just the individual or corporation or nation-state which creates it.)

> It’s unfalsifiable unmeasurable woo-woo.

An hypothesis that cannot be falsified does not count as a scientific theory (according to Popper, not everyone agrees). That does not mean that is woo. It just means that it's outside the epistemological limits of science.

Another thing that is unmeasurable is science, and yet most people don't doubt it.

> The singularity and AGI are apocalypse for atheists.

I agree with that to a degree.

Agreed. There is a really funny article by idlewords (who also regularly comments here) about this: https://idlewords.com/talks/superintelligence.htm

Not only is the Singularity religion for atheists, it also sparked tremendously bizarre offshoot beliefs, like that "Roko's Basilisk" meme that the "Eliezer Yudkowsky" crowd was briefly obsessed with.

There are several different types each of which has its own testable predictions. I can’t play the linked video, but given this is Vernor Vinge, the prediction would be “a computer which is just as good at designing chips as a human, so when Moore’s law doubles its speed, it now takes half as long to reach the next double-performance design”.

> The singularity and AGI are apocalypse for atheists.

I don't follow the logic of this statement. I'm an atheist, and while I don't believe there will ever be any "singularity", I also don't understand how the impacts of AGI would be different for theists versus non-theists. Could you explain?

I don't think this is likely to happen. There is almost surely a class of interesting questions it takes AGI to answer but a math PhD to pose.

The machine stops.

people still play chess even though machines are a class above.

I never hear about nature catching up to us- its always defeated and in the corner, we remain dominant. But as soon as society crumbles, meet those creatures and plants we couldnt get rid off- hyper adapted to resist us. Plants to which we are extremely allergic, who are resilient to all herbizides and selective bred for maximum spread into our ecologic niche. Animals, who by virtue of surviving in information dense environments, are very stealthy- and very clever. Carrying out of knowledge or instinct diseases and allergic contagions, to drive us further back into domes. Most of our companion animals/plants unable to dwell in the open in the overheated world we created. All the resources we had for the first attempt gone- no easy to reach oil, the best mines old landfills and destroyed citys.

To return from such a dark age, seems unlikely.

Or a bit of both - we'll wreck the environment thanks to AI being used as a political destabilizer (cf Cambridge Analytica).

Note this doesn't require AGI, just basic machine learning finding ways to push our buttons to make us click more buttons...

Some people like the dark ages. The advantage goes to aggression and physical strength. Some would rather compete in such a world. It could be why they seemingly battle against civilization.

This kind of pessimism is not justified, environmental conditions near most large cities are much better than were in 50s, serious issues from climate change that can cause dark age are predicted to happen only after this century, if we keep doing what we are doing now, but by that time we'll have so many new technologies, that these predictions would look as unrealistic as the predictions from 1894 that "In 50 years, every street in London will be buried under nine feet of manure".

A neuron uses far more than electricity to communicate. The number of neurotransmitters and receptor types alone is staggering. There’s a reason why biology evolved past using gap junctions. The computer science view of the brain compares how the brain works to FLOPS, much like how previous generations compared the brain to horsepower from the steam engine. Better analogy, but still naive.