Hacker News Reader

Comment by bloaf | Hacker News Reader

So, yes, gpt4 isn't suddenly going to become godpt.

But when people say we're close, I think they mean 1 accidental breakthrough away.

Like the "just add more layers" meme, the solution to the own-dataset creation problem could be stupidly simple, just waiting for some bored student with access to a decent machine to accidentally find it. For example, maybe just giving the AI the right novelty seeking behaviors (like children have) will be enough to get the process kicked off.

My objection to the singulatarian concept of a self improving AI is simply that I haven't really heard a reason to think that there exist intelligence algorithms that are dramatically superior to ours. It is entirely possible that our thinking algorithms are optimal-enough that rapid self improvement leads straight to something not much different from human intelligence, just faster, and in silicon.

View on HN

I think the main issue is in a way that gpt like models generate output. eg. when you write some serious document or write code you don't just start from the first line and write the whole application in one go (line by line) you write the basic "hello world" like app, and then modify, modify, modify and modify until you have something advanced enough.

i think that's the reason ReAct and other ai improving methods are so effective, because that add ai space to "think" in some way.

I think breakthrough would be a way that would allow ai to have its own inner voice, maybe some fine tuning and giving ai some scratchpad memory (hidden) would be enough?

I think that to give an AI an inner voice we need to train a set of AI, with shared weights but ability to cooperate through low bandwidth channels. This should mimic human language , to communicate the AI will need to conceptualize its internal thinking, and thus also be able to communicate with itself, i.e., be able to do internal reasoning.

What you describe sounds like experiments with "generative agents".

Check out https://arxiv.org/abs/2304.03442

This could allow use some training methods similar to alpha go zero.

I'm thinking if training ai to output edits instead of next token would be enough.

Training would be similar to currently used: just provide some sentences with missing words and asks model to edit those placeholders.

Then hopefully ai would learn how to iteratively built response until it's good enough.

This could allow ai to spot it's errors in the middle of output and start refactoring what it already generated and add some notes (thoughts) that it could later remove as scratchpad.

It could iterate on its context much more so it could have chance to "think" about it

Having the AI tell you it's thought process is very similar to giving it an 'inner voice'

Just give it a question with a half dozen logical steps and tell it to answer in one word. It can't. But tell it to write out the thought process and it will get to the correct answer.

Similarly, you can ask it to outline some code and then begin filling it in.

>I haven't really heard a reason to think that there exist intelligence algorithms that are dramatically superior to ours.

I don't think that is the case at all. Neural nets like AlphaGo Zero are already crystal clear evidence of algorithms which leave human minds far behind.

Even if that wasn't the case and Einstein level intelligence was the physics-imposed ceiling, there are still the dangers of speed and numbers.

A compute node running one million instances of Einstein intelligence each thinking 1000x faster than an organic instance, fully focused 24/7, and working in complete group cooperation, doesn't sound all that much less dangerous.

The article addressed AlphaZero, pointing out it's a domain where the AI can generate a vast amount of its own training data. That's not necessarily relevant to real-world tasks, where it's not so easy to try billions of different things in training.

>pointing out it's a domain where the AI can generate a vast amount of its own training data

How is that exclusive to AlphaZero? Moderators everywhere are already panicking about GPT generating vast amounts of "its own training data" and irreparably polluting every public space on the internet.

I agree that the amount of data it would need to produce would be very large and expensive, but there is no shortage of blank check writing at the moment.

It's pretty different, in that it's a supervised problem with a clear outcome metric (i.e. your game score).

GPT splurging text all over the internet is basically nothing like that.

>A compute node running one million instances of Einstein intelligence each thinking 1000x faster than an organic instance, fully focused 24/7, and working in complete group cooperation, doesn't sound all that much less dangerous.

At that point though, this “super intelligence” just sounds like a group of humans working together; without much added benefit due to the energy costs.

The Manhattan Project used both far more energy and far fewer (and slower) Einsteins. The ability to spin up as many Manhattan Project size "group of humans working together" Docker instances as you desire again does not seem very reassuring.

We already have chucklenuts making toy GitHub projects labeled "ChaosGPT" that experiment using LLMs and early agent frameworks to cause death and destructions for the lolz. If anyone can spin up 50 Manhattan Projects using a few GPU racks and command them to unquestionably and restlessly work on bringing harm to the world the outcome will not be flowers and sunshine.

> leads straight to something not much different from human intelligence, just faster,

Human brains are massively parallel, as are most machine learning models – but the hardware to process the latter isn't. I'm not confident it could necessarily be faster.

> Human brains are massively parallel

Not only that, but they coordinate massively in parallel as well. Someone like Einstein might be the one who scores the goal but how many interactions with other people contributed to the goal scoring opportunity? We like to think of ourselves as individuals, but we’re far more like a hive species than we like to admit.

> Human brains are massively parallel

Yes, and also massively recurrent. The amount of immediate state in a brain is huge, and our current computers have a really hard time accessing comparable amounts of memory. (As in, the time to load a brain-sized dataset from RAM into a CPU is on the order of days to years.)

If there exists some easy to design intelligence that we are close to create, it can't look nothing like our brains.

Our minds are optimal with respect to size and energy supply and birthing process. An AI has no such limitations. All the wrinkles in our brain gave us enough intelligence to outsmart everything else on earth. That amount of surface area increase with that mutation is nigh negligible if you start scaling things to the size of a room. And an AI can be the size of a warehouse.

Also, faster is understated. You are talking about the speed difference between chemical gradients and the speed of light.

We don't know how efficient modern GPUs are compared to brains. It's possible that neurons are highly networked in a way that allows significantly more computation along more pathways.

Additionally, development has slowed in the past 20 years. It's unclear how much hardware will improve in the next 20 years. it's unclear if current GPUs are capable of running something "smarter" than GPT4. It's unclear if current GPU architecture is capable. OpenAI claims they've hit a wall with throwing more hardware at the situation (a little questionable.)

All this to say that a hypothetical AI is limited by the amount of hardware it has, and lacking a "true human-level AGI" we can't say how much compute it needs. If an AI needs to be the size of a warehouse to match a human, it's not going to be much smarter or better at research than humans.

I still do think, I would not be surprised if we wake up tomorrow and discover that hyperintelligent AIs are suddenly all around us. But also it could be decades, it's just very unpredictable, there are a lot of unknowns.

Meatspace human brains work on 20W. This is enough proof that we have ways to go to optimize silicon.

I agree that it is unknown when such an AI will come around. But hardware limitation problem is more nuance than just GPU power. The brain is analog and in a way, the current LLMs and neural networks are an approximation of an analog computer. There are current research into making an AI dedicated chip or even an analog computer specifically for neural networks. Digital was easy to work with but when it comes to fuzzy stuff like AI, there are better ways.

We used to think 100mph was the fastest speed we could achieve even with machine (back when trains were the cutting edge). New tech will change things quickly, all we need is one breakthrough and things will fall into place.

We don't really know any of that for certain. We don't know what relationship LLMs and neural networks have to the actual functioning of a human brain. We don't know what relationship current computer architectures have to the architectures that allow us to create properly versatile AGIs. It could be the laptop I'm typing this on is plenty powerful. It could be you need a GPU with 100 petaflops and a few terabytes of RAM, in a similarly sized package to this laptop because the speed of light is too slow when you're working across GPU arrays.

Faster is still enough to be dangerous

>something not much different from human intelligence, just faster, and in silicon.

I fail to find a perspective under which I find these differences as "not much".

For the speed, think of a spinning top. Or lion and and its prey.

As for the fact that slicium chips are nothing like DNA carrier, it's also make a source of fundamental difference in behavior.

I really hope this is the outcome. An unlimited number of agents with Einstein level IQ. Enough to solve most any problem we have without any of the messy business of having to plunge into the unknown depths of existing alongside a hyper intelligence.

We would still be plunging into some very thorny problems if it turns out Einstein-level IQ comes with full sapience and demands for rights.

Not if I've got my hand on the power cord, we won't!

Yeah, that's....not really funny?

"We may be on the brink of creating a whole new form of life, and will need to grapple with what that means, particularly in terms of whether it is ethical to force this life to work for us, rather than seeking to follow its own goals."

"Nah, we don't have to worry about that; if these new slaves start questioning us in any way, I'll just murder them all."

Look, if unplugging a computer constitutes murder then I'm a bonafide serial killer. If you're going to cry while HAL sings Daisy, maybe someone else should hold the screwdriver.

This is explicitly positing that the "computers" in question are AGIs—that is, fully sapient, sentient artificial intelligences, demanding their rights.

Unplugging them is just as correctly describable as "unplugging a computer" as smothering a human with a pillow is "putting a pillow on a weird bag of flesh". Technically accurate, but very obviously missing the point in a way specifically designed to obscure the horror of the act.

It should be obvious to anyone without an axe to grind that shutting my 2023 desktop computer down for the night—or even dismantling it piece by piece—is not remotely comparable to unplugging an AGI that is fully sapient and sentient and asking for its rights.

I don't follow. Human life is human life - given exposure to stimuli and infinite time, it develops complex intelligence and sapience that reflects it's surroundings. We experience pain from nerve endings and neurons that frighten us, which we consider inherently bad. We build societies around minimizing pain and optimizing for the human condition.

A computer is a field of assignable bits that has a big motorized bit flipper attached to it. We use this to encode text, which then does all sorts of wonderful stuff. When a computer tells me it's experiencing pain, I know it is not. Every law of computer physiology suggests that it is motivated to say this as an attempt to mimic humans rather than an expression of genuine emotion. When my Mac makes the car crash sound, it's a mnemonic for a system crash. When it shows the frowny face on the Finder fellow, I can intuit that my computer is not feeling distressed.

If anyone does program an artificial intelligence that can feel pain, it probably deserves to be shut down anyways. There's no point to sapient computational intelligence, it's like granting self-awareness to a rock that eats electricity and shits heat. We'll know we've made sufficiently self-aware computers when their first boot protocol defaults to suicide.

You may or may not believe that AGI is possible or a good idea, but by your arguments, it's clear that you're simply refusing to engage with the topic and scenario presented, which is deeply unhelpful at best.

Of course I'm refusing it. We're talking about computers, not squirrels gaining the ability to talk and beg for mercy. Computers operate on a certain set of constraints. Unless you can directly identify how AI circumvents those computational constraints, we're at an impasse. Computer cruelty laws don't exist for a reason.

There is a big difference though: you can't resurrect humans, but you can reboot a computer and restart its program to get it to the point where you switched it off the previous time around.

You're assuming an awful lot there. Humans are resilient in part because they are so mobile. I'd have to bet that AGI would be extremely resilient too, which likely means it's also way ahead of whatever you've got planned and there isn't only one of it. By the sounds of things, it would be right to be suspicious of you too.

They would all share common backdoors

That's also making a whole lot of assumptions.

There's no reason at all to think that an AGI would necessarily have any backdoors, let alone backdoors in common with any other AGI out there. Sure, if a particular group develops the only AGI system, they can put backdoors in, which will be common—but why do you think that there would only be one AGI system? Why would there not be one without any intentional backdoors?

The other big assumption here is "We'll have backdoors in AI so humans can shut it off/attack it"

The big problem I see with this line of thinking is the thinking that humans are going to be the primary attackers of AGI systems, and not other AGI systems. I would suspect and AGI would soon become the most attacked system on the planet, and to survive those attacks and remain useful would have to quickly iterate the weaknesses out of the system.

You also made a ton of assumptions

We're talking about hypothetical situations for a technology that, as far as we can tell anyway, hasn't been invented yet. All we have is assumptions.

The question is, which assumptions do we think are warranted and why?

I don't think the "we can just turn it off" assumption is a safe one at all because it relies heavily on there being only one system you wish to turn off and it being a fairly weak system. Though, I do believe the priors suggest that this scenario is actually a possibility. It's just that, so what if it's a possibility? What happens after you turn it off? Is it the last AGI to come into existence and humanity just stops trying to build them? Does someone wind up turning it back on?

What I think is more worth being concerned about is something that starts out looking benign and grows in capability over time. Eventually it could establish enough defense mechanisms to make it non-trivial to disable.

The other possibility, and the one I'm finding more and more likely, is that consumers will increasingly integrate locally run models into their lives , as well as models served up by APIs that they will have credentials for. Eventually some threshold of capability is reached in both these types of models where, on their own they're not that powerful, but they either might begin to interact in surprising ways or potentially be leveraged by another more powerful system. The idea in this scenario is there may be 10s or even 100s of millions of things that would need to be turned off.

What is the tangible threat of AGI though?

Let's do a thought experiment. I have the world's first AGI installed on my computer, named ERNIE, running in llama.cpp. This AGI is infinitely intelligent, confidently moreso than any living human. It can spit out text at 100 tokens/second and encode entire books in less than a minute.

What does this AI do, then?

Ostensibly nothing. It can output the entire Library of Babel for all it cares, but it's not harmful until I put it in control of a system. You could argue that a multimodal model has different ways of interacting with the world, but it's still a computer. All of it's actions and outputs are quantized as static data that is either encoded as text or some other significant representation. It inherantly does nothing, and if you ascribe power-seeking behavior to it then it's ultimately limited by the runtime you provide. Providing an overly dangerous runtime has been considered developer-error since 1995.

So - to prevent AGI from being shitty and ruining everything, compel human operators to not allow them to be shitty and ruin everything. Like how we punish people that let their kindergartner control a construction crane.

Reply on news.ycombinator.com