Why don't governments mind that companies are explicitly trying to make AGIs?

I think we are getting the wrong signal from governments lack of interference in this field, "interference" being important.

Governments know that AGI will be a game changer if created, and governments are in the business of countries. They know that if it is going to happen someone is going to do it eventually, might as well be someone that is a citizen/registered as a corp in their jurisdiction. Why would the US for example want to stop Google from achieving AGI supremacy? So that Tencent can do it first? They're stuck, AGI would be a threat against their power, but stopping it will ensure that a competing country has sovereignty over the creation instead. It's not incompetence, it's a legitimate conundrum, it has elements of MAD and public/private power balances and geostrategy.

Personally I don't think AGI is possible. I think there's some element to intelligence that is not just some threshold of complexity, maybe it's some architectural feature or perhaps something more fundamental. What I think might resemble it in some ways, and IMO is the future, is complex application specific computation. Imagine an algorithm that can take over for example nuclear strategy decisions, but can't really tell you the most optimal layout for roadways, off the top of my head of course the two problems might fundamentally be the same optimization problem, I don't know, but it's an example. Powerful machines that do one thing really well, not one system that does everything well.

I’m surprised at the amount of comments greatly downplaying the likelihood of AGI in the coming decades or saying that is impossible.

The rate of progress is AI technologies seems absolutely incredible to me. Sure, there is tons of hype and noise to weed through. And much of the technology and research hasn’t yet found commercial applications. But the progress in recent decades is objectively incredible.

We are reaching a point where all games are basically falling to AI. Go, no-limit hold ‘em (poker), StarCraft, Dota, and so on.

In other domains, we have GPT-3 and AlphaFold. And I’m sure there are many developments I’m not aware of.

From what I can tell, GPT-3 is mainly significant for challenging that notion that increased scale can’t be the primary factor in a significant AI developments. The jury is still out, but my understanding is that it demonstrated that modest tweaks to existing algorithms with a massive increase in model size can result large performance improvements. GPT-3 had quite limited “memory”, which is one of the reasons it struggled with coherency. How would a model 10x - 100x larger than GPT-3 with a significantly enhanced memory perform? What if it was trained to predict the next frame in video in addition to the next sequence of text blobs?

No one knows how close we are. It will likely happen quickly without many people knowing it is coming soon. It will seem “impossible” to most even as the last few pieces are falling in place. Major technological paradigm shifts usually happen unexpectedly, as history shows us.

I’m rambling a bit (If I had more time / skill I would have written a shorter comment ;). Suffice to say I’m surprised by the number of confident proclamations against AGI. Our brains are amazing. But they aren’t magic.

This question is easily dismissed because the author is asking about something that is not likely to emerge in the near future. A more interesting and less easily dismissed question would be to substitute 'game changing technology' for AGI. I think there are a number of technologies which, while not AGI, provide a significant edge for a nation or other organization possessing them.

Biological technologies. Energy storage technologies. Financial manipulation technologies. Advanced propaganda technologies(think 'deep fake', or deep fake combined with mass surveillance and communications infrastructure penetration(i.e. man in the middle a society's communications and start a mass panic)). There are probably many, many more.

People worry a lot about AGI, but all that's required for, say, killer robots is a solution to the power problem. We can already mass produce drones with good enough pattern recognition to slice your jugular. If you figure out how to power a few million of those drones for a day or so, you have one of the most lethal and targeted weapons in the history of mankind.

I think it's more likely that the robots won't kill us though. It's far more likely, assuming such a doomsday scenario takes place at all, they'll just convince us to kill each other or ourselves. Perhaps we won't even have to die, we'll just need our priorities adjusted to serve their owners.

To return to the original question though, I think there is a naivete amongst technologists today regarding the level of integration between corporations and the military. Si Valley basically exists as an arm of the military industrial complex. Anyone who thinks the government isn't funding and making use of the latest technologies in processing, pattern detection, deep fakes, deep learning, etc is simply ignorant of reality. These things aren't talked about much because they're upsetting to younger engineers.

AGI is a pipe dream. It certainly is a moonshot goal to aim for, like long-term population of Mars, but the likelihood of achieving that goal in the short term is about the same as the likelihood of a sustainable population on Mars in the short term. I wonder which would happen first (or at all). I wonder honestly how much the chase for AGI is about the end goal and how much is really about money.

The concept of AGI is extraordinarily materialistic. Most humans through history believed in spiritual dimensions of reality, including many of those responsible for developing technologies we use today.

The argument for AGI goes something like, the brain is a self contained computational machine, humans do extraordinary things with it, therefore a turing machine can achieve human+ level competence.

Most humans, who are not materialistic, see the brain as an interface to both physical and spiritual realities. Computers cannot access the spiritual realities, except indirectly through human input.

I'd wager that we're more dumb terminals connecting to the mainframe than most of us would like to believe.

Ah yes, an AGI "superintelligence" piece. People who read too much scifi worry a lot about these apocalyptic AGI because they have some idea that an abstract "superintelligence" can go on self-improving exponentially using nothing but its own smartness, leaving people behind in the dust.

Curiously absent from this picture is the part where the AI is a physical thing installed somewhere. I'm sure the newborn superintelligence will immediately set about beating chip-makers like TSMC at their own game by merely imagining a better source of extreme ultraviolet for lithographic processes than the current laser-against-molten-tin solutions, to say nothing of the cleanroom manufacturing processes, the ultra-pure silicon, the various acids and even just supplies of water [1].

Meh. Worry first about firms and governments who operate computers who have good enough computers that they can spin up regular-level intelligences on demand. That's going to change more games a lot sooner and a lot faster.

[1] https://fortune.com/2021/06/12/chip-shortage-taiwan-drought-...

Can't remember who said it, but worrying about AGI is like worrying about overpopulation on Mars.

The same reason that religious generals in the military don't take into account the possible interference of their god in their war plans. Of the subset of people that intellectually believe in AGI most don't actually believe in a way that effects their actions or planning.

AGI is a weird term because it seems to mean different things to different people. Generally it seems to mean "parity with human cognition", but cognition is not just one thing and some ML models already outperform humans in narrow tasks.

A lot of criticism of AGI is more aimed at ASI (artificial sentient intelligence) a sentient, conscious entity with goals and agency. We haven't made much progress on this front, but there definitely has been progress on AGI in the sense of general intelligence - ML models that incorporate many modalities and generalize to diverse tasks instead of a single narrow task. Such a model need not be sentient to outperform humans on economically important tasks, and we're rapidly heading in this direction right now.

You have to remember that just because a thing can theoretically be achieved, doesn't mean that we have the motivation to do it. We still don't have any planes as fast as the now-shuttered SR-71, and that thing was developed 58 years ago. We have had electric cars for nearly 200 years, and they are barely starting to become mainstream. The technology for an elevator has been around for thousands of years, but it took tall buildings becoming more common for somebody to both imagine a simple enough mechanism, and see the market for it. There has to be a large enough will and force applied for technology to become reality, and even then it might die on the vine.

AGI would only ever be possible in our lifetime with a new Manhattan Project, and even then it would take sooo much longer than nuclear science. Commercial software development today is just laughably bad. Only slightly better than it was in the 1970's. The only thing we have going for us today is processing power, and it's still at least a century away from having the kind of raw power to brute-force compute the kind of probabilities to even accurately predict the weather. Reproducing the cognitive ability of a 5 year old is still a very long way off.

It would actually be significantly faster to develop AGI by just mapping a brain. Sort of how autonomous driving is developed today; just run the stupid car on the streets for a billion hours and end up with a gigantic statistical model that you smooth out by hand, rather than trying to make a pure algorithm that can drive correctly in any situation.

Because the government entities who mind this issue are different than the ones you are looking at.

https://www.nato.int/docu/review/articles/2021/10/25/an-arti...

Because it isn't going to happen?

I'm highly confident they are paying close attention to these companies.

Because everyone at the top levels of government has the security clearance to know that The Basilisk is real. This is also why Presidents age so visibly and dramatically in just 4 years

It would be great if people could start by defining AGI before engaging in a discussion about it. What is AGI? If I asked what fusion power is someone could do that, down to the mathematical level. At best if asked what AGI is you’ll get a vague cobbled together anthropomorphic analogies.

It was Locke who said that all arguments come from a disagreement about the meaning of words. That’s exactly what’s happening with the word “AGI”. It’s pointless even talking about it if everyone can’t agree on what it means.

Look at Y2K, climate. Government won’t do anything until it’s closer to real. When it’s nearer you can bet they’ll get focused minds, fast.

Non-zero chance they drive in and take over. As a government who insists on holding the final power cards (eg violence) you can’t take chances with this because government power means nothing in the face of a significantly smarter entity. Instantly you wouldn’t be able to trust any electronic communication. Not news, not email, not websites, not IM.

why does the author presume that governments (or at least the interested bodies within them) aren't fully appraised of the status and utility of each of these projects?

i don't think anybody's close to "agi", and anyways it's far more profitable/effective to just use more specific tools to accelerate the structures and processes that already exist.

if anyone was "close" i don't think they would be very public about it, but i would expect intelligence agencies to be aware. in that case, if some particular project was perceived as an existential danger, you could expect them to act appropriately.

everything the author seems to fear is not coming out of "agi" but instead much more mundane tools that are already proliferate. i think we're going to see the benefits and destruction from the technology we already have surpass anything any of us can predict before anything like "agi" nears completion.

Governments don’t create companies. Companies create governments. (Governments don’t create people, people create governments).

Companies trying to make AGI, is just people trying to make AGI.

The limits of intelligence are more a limit of energy and resources. I mean there is plenty untapped human intelligence.

If this is a thing people can actually build, government would have a hard time stopping them, or even knowing that people were working it if they decided to keep a low profile. It might not even require specialized hardware, or a particularly large software team.

one possible answer: the people in government tasked with minding such things believe (I think correctly) that the [fill in the blank]/industrial complexes at the heart of their operations are about as close to AGI as we're likely to get for some time, barring unforeseen advances

in other words the AGI that matters in the near term has humans at the core of the loop

Probably every company with credible AGI ambitions has one or more spies planted by one or more countries.

They really ought to keep an eye on that Alcubierre guy, he invented a warp drive. /s

I think it is because they, rightly IMHO, don’t actually believe that it is possible.

People like to mention that the brain has billions of neurons and how that is completely out of reach for current technology... But that neuron count includes neurons from brain structures related to non-cognitive tasks such as controlling your heart rate, breathing, glands, etc. Also, not everything in a neuron is for information processing, part of it is responsible for keeping the neuron alive, and part of it has to do with communication at the lower levels of the OSI model.

With respect to a theoretical artificial intelligence, we have very deep disadvantages that people do not talk about often.

Verbal communication in humans, seen from an oversimplified perspective, is about encoding ideas into words and then emitting the respective sounds for those words with your vocal chords and mouth, so that then another person listens to those sounds to recognize the words, and then inferring the meaning for each word and then the phrases. To make verbal communication more efficient, new words can be created to represent high level concepts, making faster communication possible. But there are still limits to verbal communication, both as speakers and listeners.

But what about machines? Machines do not even need to translate their thoughts into words. Machines can simply share a serialized cognitive model with another one, at gigabytes per second, and then the receiving machine can load it. What does this mean? that for machines, going to school is a experience that will last only a few seconds, and after that machines will be able to pass every school exam with 100% score.

Once one machine achieves expertise in a field, that machine can share its model with others, who will then load it and become experts immediately. This means that once one machine becomes as smart as Einstein, few minutes after, millions of machines will be as smart as Einstein.

It is not really the fact that we will be competing with smart machines, but the fact that we will be competing with machines that do not take 18 to 25 years to become "productive" in the job market, and that can educate themselves at a ridiculously fast rate making it impossible for us to catch up.

Also, the fact that machines can simply forget and throw away a model and incorporate a new one as needed. In a machine economy, AGI machines may be able to switch jobs multiple times per day, redirecting their effort to meet the needs of the economy in a highly dynamic way. They'll never have unemployment because they will quickly readjust their skill set to what needs to be done.

And when it comes to war, once machines have one highly successful general skilled at military tactics and strategy, every member of the robot army will be potentially as smart as that general.

Then, not only computer science, AI and ML are advancing fast, but we are also at the same time reverse engineering biology. In Terminator II, the fictional company Cyberdyne reverse engineers a robot arm and a robot chip from the future. That's not necessary: the "future" technology is already here. The robot arm = human arm, and the robot chip = your brain.