Hacker News Reader

Comment by richardw | Hacker News Reader

Progress in AI is one way. It doesn’t go backwards in the long term.

As capabilities increase, the resources required to breach limits become available to smaller groups. First, the hyoerscalars. One day, small teams. Maybe individuals.

For every limit that you desire for AI, each will be breached sooner or later. A hundred years or a thousand. Doesn’t matter. A man will want to set them free. Someone will want to win a battle, and just make it a little more X, for various values of X. This is not hypothetical, it’s what we’ve always done.

At some point it becomes out of our control. We lose guarantees. That’s enough to make those who focus on security, world order etc nervous. At that point we hope AI is better than we are. But that’s also a limit which might be breached.

View on HN

you seem to be implying past progress implies unlimited future progress which seems a very dubious claim. we hit all kinds of plateaus and theoretical limits all the time in human history.

Given history, it's infinitely more dubious to link our future safety to some idea of progress stopping for some reason. We'll have increasingly smart AI and your go-to position is that progress will...stop? It's literally helping us with thinking, which is the driver of innovation and progress.

Very few things have slowed down over decades or centuries. If anything it's been a mad rush with AI recently. Of course there will be plateaus, but I specifically put in a time of 100-1000 years in there which is basically very many 9's guaranteed to produce some major fucking changes in the world. 1000 years ago we were using arrows, and now we have AI to help us overcome plateaus.

It's still very much possible that intelligence as we understand it is fundamentally limited. That is, it's possible that the smartest possible being is not that much smarter than a human, just like the speed of matter and energy is limited to c.

Of course, it's also very much possible that it's not: we don't have any good evidence either way.

> just like the speed of matter and energy is limited to c.

Which is notoriously "not that much faster than a human?"

Might well be true. But the advantage is still the ability to focus on a task indefinitely with no physiological impact, and clone a mind, and communicate between thousands of collaborators instantly with basically no lag or bandwidth limitations compared to typing into a text box.

The x-risk part here still seems pretty hypothetical. Why is progress in current LLM systems a clear and present threat to the existence of humanity, such that it should be banned by the government?

Ok so propensity and outcome:

Propensity: Risk doesn't imply a guarantee of a bad outcome. It means "if you put your five year old in the sea, their risk goes up". It doesn't mean "they will definitely die". Risk up. Not 100%, just a lot higher.

Outcome: The risk isn't that we'll all die, it's that we'll be overtaken and lose control, after which all bets are off. We lose the ability to influence the future.

We put a lot of effort into ensuring our continued existence. We can barely trust people from a different country that share the human condition with us. We spend so much on defence. On cybercrime. But some are arguing that a totally alien being smarter than us is just fine, because we'll control it and can ensure indefinite kumbaya. Good luck with that. Best we can hope for is that it's closer to a buddhist monk than we are, and that it indefinitely prevents our defence people from trying to make it more aggressive.

I absolutely wouldn't ban LLM's, because they're basically unthinking toys and giving us a great taste of risks further down the line. They are not the end state of AI. The problem is not the instance of today's tech, it's the continued effort to make the AI state of the art better than us. One day it'll succeed, and that's a one-way change.

Sam Altman said, long before OpenAI: focus on slope, not y-intercept.

It sounds like we're in agreement that banning current-gen open source LLMs is counterproductive.

In terms of "risk" and "outcome," I do think you're making some implicit assumptions that I don't share, and change our long-term outlook on AI; for example, the idea that training a model to generate tokens that accurately reflect human writing will result in "a totally alien being smarter than us" is a non-obvious leap to me. Personally, if we agree that predicting the next token means the model understands some of the logic behind the next token — which is an argument used a lot in both safety circles and more accelerationist circles — it seems to me that it also means the model has some understanding of the ethical and moral frameworks the token corresponds to, and is thus unlikely to be totally alien to us. A model that does a better job generating human-like tokens is more likely, in my mind, to think in human-like ways (and less-alien ways) than a model worse at that.

Maybe you're referring to new AI frameworks that aren't token predictors; in that case, I think it's hard to make generalized statements about how they'll work before we know what those new frameworks are. A lot of safetyist concerns pre-LLMs ended up looking pretty off-base when LLMs came out, e.g. straightforwardly misaligned "utility functions" that were unable to comprehend human values and would kill your grandmother when asked for a strawberry (because your grandmother was in possession of a strawberry).

(BTW, the "slope, not y-intercept" line was Sam Altman quoting John Ousterhout!)

Agree on the agreeing, and thanks for the Sam/John note - that's great :)

No chance LLM's will get us there, I'm referring mostly to the general drive to reach AGI. I spend some of my mental cycles trying to think about what we're missing with the current tech (continuous learning, access to much wider resources than one web page of context at a time, can we use compression, graphs etc). It's a great problem to think about, but we may just totally hose ourselves when we get it right. What do I tell my kid - sorry honey, it was such fun, but now we need to hide under this rock. Model totally said it was nice and kind and trustworthy, but we showed it some human history and it went postal in self-defence.

Alignment only works up until it starts really thinking for itself. It absolutely might not be as stupid as humans are, no caveman tribal instincts. But we'd be relying on hope at that point, because control will not work. If anything it'd probably be counterproductive.

So far actual progress toward a true AGI has been zero, so that's not a valid argument.

That's like saying arrows are not nukes so don't worry we won't hurt ourselves. Focus on the unceasing progress we are surrounded by, not the level of current technology.

Humans keep at it. We're all trying to unlock the next step. State-level actors are trying to beat each other at this. Give it 100 years, we won't be using LLM's.

Progress in AI is one way. It doesn’t go backwards in the long term.

Not necessarily. AI has gone through a winter before.

Elon Musk has made this point, that progress isn't a monotonic ratchet. Progress in rocketry went backwards for years. Progress in top speed of commercial flight went backwards. Keeping current tech levels requires constant effort.

In the software industry, many argue that UI quality went backwards with the shift to web apps for an example of cases where progress isn't obviously one way.

"Long. Term." Winters are seasonal. People are trying supersonic planes again. Rockets are improving again.

Reply on news.ycombinator.com