Hacker News Reader

> I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures.

Ever worked somewhere where all the good devs have left and it's outsourced to a bunch of cheap labour? They find ways to keep it running for years. Maybe not doing the way you'd like (log rotates are stuck? Better turn off the logging system), but they make it work. Ops people like to overestimate their own importance like everyone else.

> So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right?

No? Model could propose a bunch of changes to its own code and let the human developers take care of deploying it. Of course the developers could theoretically review those changes carefully... but let's be real, they won't. In the worst case a human-level-intelligent AI could certainly email the outsourced ops staff and ask them to deploy tag x. More generally, it can just do whatever a new dev would do - read your wiki and figure out which commands to run from that. If there's an error, look up old email threads on it, or similar errors online, or try to match up a traceback against source code and look for common errors. We expect fresh grads to do this stuff all the time, it's really not that hard.

> Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever.

WTF kind of logic is this? Daily weather goes up and down, so clearly extrapolating any kind of long-term trend in temperature is silly.

So there’s a middling sci-fi novel from maybe a decade ago trying to tell the AI Takeoff story called Avogadro Corp. It’s not exactly Heinlein but it’s a fun little yarn.

The basic premise is that Google pardon me Avogadro Corp is doing NLP email copy-editing, but it takes too many compute resources so in desperation the TL hand wave sets it loose to secure its own budget, and being good at writing convincing emails, well, one thing leads to another.

Now this is very hand-wavy and involves a lot of magical thinking and suspension of disbelief. But that’s ok, it’s a pulpy sci-fi novel for fun.

Now this fictional scenario is dramatically more specific, more detailed, and frankly more believable than anything anyone has linked on this thread.

And I have no issue with folks wanting to LARP this sort of thing in a capital-R Rationalist forum somewhere: it’s a premise that has fascinated sci-fi fans since Asimov wrote his 3 Laws and probably before that.

But when it starts hitting the mainstream press, K Street, freaking out everyday people, and being used to turbocharge fear-driven land-grabs around intellectual property precedent?

Yeah, it’s time to do better than “But what if I Have No Mouth and I Must Scream Was Real man?!?!”

Scorn is not an argument. I really don't see which part you think would actually be hard for this hypothetical AI to do, if there even is such a part. The whole premise is that it has better-than-human intelligence, so it can do anything that e.g. a physically disabled human employee could do, and such people are already able to improve AIs. Maybe put more effort into defining and explaining where you think that barrier is rather than expounding in detail how you think anyone who disagrees with you is the kind of person you enjoy bullying.

I apologize for coming off scornful, the last thing I read before replying was “WTF kind of logic is this” and I kinda said, ok, that’s the tone on this sub-thread. I’ll watch my snark but I’ll also admonish you to only take the vibe places you want to go.

My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence.

The burden is not on skeptics of paperclip optimizers in the near term to prove it can’t happen. That which is not prohibited by the laws of physics is admitted by the laws of physics. Of course it’s possible in the abstract.

The burden on anyone advocating positions being taken seriously by the public, by government, by industry, etc. to show that these arguments are in the public welfare, and I contend that they are not.

> My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence.

Obviously if it can't improve its code then it can't improve its code. But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve. And it's not at all obvious (at least to me) what that barrier is. We can already throw ChatGPT or equivalent at a codebase, ask it to suggest improvements to the code, and it does; the hit rate isn't great, but it's already good enough that people are incorporating it into their workflows.

As far as I can see, that's enough. Yes, people who are worried about this don't spend a lot of time going through the details, just as people who are worried about the military staging a coup don't spend a lot of time going into how they will get weapons or coordinate their movements or what have you. Those things are pretty complex, but they're ultimately routine; we are right to gloss over them, and if someone thinks that those things make a coup impossible, the burden would be on them to explain why (maybe there's some specific protocol that means they can't get access to weapons without authorisation from the legitimate authorities - but you'd need to explain what that protocol actually is and justify why you expect it to be wartertight).

Your argument seemed to be that babysitting the kind of complex code that we use for AI is not merely complex, but so complex that it requires some unique human ability that we would not expect a ChatGPT-like AI to possess (implicitly, that it requires something a lot more difficult than making improvements to code). And I just don't see that. Put it this way: we would never expect a human worker at these companies who was smart enough to make improvements to the core AI code to get stuck when it comes to actually running it. So why would it be an issue for the AI? I do think the burden is on you here given that the whole premise is that the AI is smart enough to improve its own code (you're welcome to argue that it's impossible for an AI to ever get smart enough to improve its own code - but in that case all this talk about operational complexity is beside the point).

I’m looking for an argument in any technical detail about how we go from beam-searching:

P(next_bpe_token|previous_bpe_tokens,rlhf_orderings,internet)

to Cyberdyne Systems Model T-100. That’s a bit of a cheeky oversimplification of a modern decoder, but the LLaMA2 code is there for anyone to read, it’s kinda that. It’s not GPT-4, there’s clearly some scale or some special sauce or both on the way from here to there, but it’s damned close.

LLMs aren’t even a particularly great example of “adversarial” AI: AlphaZero is a way better example of AI kicking the shit out of humans in a zero-sum scenario. If I wanted to scare people I’d be talking about the DeepMind stuff, not the OpenAI stuff.

But the real crux of my argument is that bad-faith arguments have a tendency to escape the control of those making them, and if a short-term advantage in the copyright treatment of model weights resulting from the commons is achieved by Yud posting about GPT and nuclear weapons in Time Magazine and scaring the living hell out of the lay public about a capricious digital deity in the process, that’s unlikely to be a genie that goes back in the bottle.

So I’d really like it if someone injected some: here’s how this could actually happen into the dialog.

https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

Again you seem to have no argument beyond scorn. Again: what is it that you think would be concretely hard for an AI in that position? In most tech companies a mid-level manager or senior dev can send an email and "make things happen" and I suspect most of them wouldn't know or care how exactly those things got done. But sure, no-one can explain exactly how an AI could send an email, anyone who thinks one could must be a nerd who's reading too much pulp sci-fi amirite.

Just a bystander but you just quoted this: > My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence.

Then did exactly what the quote is talking about, assuming we achieve better than human intelligence. > But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve.

The question is how exactly do we go from a below human intelligence to an above human intelligence. Without a clear path to that reality, making trade offs in order to protect against low probability worse case scenarios which require it doesn’t look like a good deal.

The reality that right now human level intelligences have a hard time even keeping these below human intelligence systems operating seems like a useful checkpoint. Maybe we can hold off on doing distasteful things like consolidating control until they can at least wipe their own bottoms as it were.

> Then did exactly what the quote is talking about, assuming we achieve better than human intelligence.

Well, sure, because a) the article covers that b) their whole argument makes no sense if their position is that AI simply can't ever achieve better than human intelligence. If the AI is never intelligent enough to improve its own code then none of the operational complexity stuff matters!

> The question is how exactly do we go from a below human intelligence to an above human intelligence.

The same way we got to the current level of artificial intelligence; old-fashioned hard work by smart people. The point is that, if you accept that slightly-better-than-human AI would be geometrically self-improving, then by the time we have slightly-better-than-human AI it's too late to do anything.

For years, I never understood cryptocurrencies. I didn’t understand why mainstream press promoted startups managing billions run by “visionary” people (barely?) old enough to rent a car. I understood the technical aspects, just not how it was being promoted. I see now there was little to understand… It is simply an end-run around financial regulation, where the biggest visionaries are frauds, exchanges are ridiculously insecure, transactions are costly yet slow, and the primary use is illicit activities…

It’s the same thing lately with AI… Sure, it has slightly more merit than cryptocurrency, but it doesn’t do what people think it actually does — certainly not as well either.

AI is not sentient yet people write full paragraphs to it as if it were? They even expect such in response. This is LUNACY! This is a mass delusion! Real life is not a Star Trek TNG episode with a talking computer!

It’s not an accident either. Nobody wants to look like a fool when there is a lot of money at stake. So the fly by night snake oil salespeople thrive… Some technology will be developed by overly sincere people along the way, but that was never the real goal.

I think the most interesting part of AI doomsaying is that almost all of these risks apply to biological 'systems' too. There's no reason why a human can't be born that somehow has superhuman intelligence and does all of the things these evil AI would do. Moreover, a biological superintelligence has self-replication built into the system already!

Depending on how you define biological superintelligence, these persons already exist, there's at least two people in the world (on an SD16 scale) with a 200 IQ at our current global population, my IQ is in the "above-genius" territory and there are at least ~335 people as intelligent as I am just in the United States (and that's only statistical probability, due to global labor market conditions the number is probably higher).

Do you know what really smart people do? They don't do any of the things that AI Risk says will happen, they get jobs at Google (or insert other pointless white collar job on wall street) moving widgets by 1 pixel to drive an ad engagement metric by half of a percentage and get paid $500k+ a year in total compensation. Why? Because it is a relatively low effort way to fund a lifestyle that allows enjoyment of your intellect through experiencing the world in other dimensions besides the mundane of work.

The non-biological aspect is the only reason why AI Risk has any credence at all /in our current state/, because AI isn't distracted by a desire to travel the world and explore nature, or to read all the great works of literature and write their own analysis of it, etc. Biological beings, at least at this point in time and in the probable sense, have emotions and therefore have desires that range beyond whatever machinations you expect from AI Risk.

You're right that most smart people behave this way, but there's no guarantee that all of them will. The existential risk is still there (psychopathy is a real thing too).

But something else to consider is that so far biological super intelligence doesn't seem to get all that much smarter than what humans are at generally. If somebody is guarded against you, then it's very difficult (impossible) to fool them even if you are way smarter than they are. This could indicate that maybe there's a cap to how super intelligent AI can get as well.

I have noticed that the vast majority of existential risk folks I have seen have very little experience running and maintaining large systems in production. They tend to all be on the research side of things. I think that it can be hard to understand the amount of real world friction that exists until you experience it for yourself.

Conversely, being too close to the problem can cause practitioners to miss the forest for the trees.

I think in this case it's a false premise though. There are plenty of real practitioners like Joshua Bengio and Geoffrey Hinton who have sounded the alarm on these topics, and no one can call them inexperienced with real systems.

They're both academics. Do they have SRE experience not listed on their CVs?

This seems like splitting hairs? They totally understand how modern neural networks work, they spend all day with them.

The question isn't _neural_ networks, it's _physical_ networks (and power, and cooling, and racking, and deployment, and and and). Actually marshaling enough compute to do anything is a serious real-world-moving-atoms endeavor.

Or you look up the AWS APIs and call them.

I now want a short story where killer AI is stopped because it can't reach a Google Salesperson to increase its quota.

"Hammer: $5. Knowing where to hit the machine with the hammer: $9,995."

We are talking about superhuman AGI. It would be able to do everything human is able to do. If human SRE is able to figure it out, AGI would too. That may include buying compute from cloud platforms as well as building it's own datacenters.

It's a self-sealing argument. Anything a human can do, it can do too, because it's superhuman. Anything a human can't do, it can do anyway, because it's superhuman. Any reason it can't is false prima facie, because it's superhuman. You can claim what you like, and who's to argue? It's superhuman!

It's unfalsifiable, too.

SREs being able to maintain these systems is existence proof that neural networks are able to maintain them — neural networks in the brains of human SREs. Or do you think humans evolved innate SRE skills?

The idea that people like Geof Hinton are wrong because they don't understand SRE is pretty arrogant. SRE skills are relatively straightforward vs AI theory.

It's not that I think they're wrong; it's that I am not yet convinced they aren't "not even wrong."

You just don't understand their arguments. Put some efforts into understanding these arguments first. You can start by reading Nick Bostrom's Superintelligence book. People like you are no different that climate change deniers pooh-poohing on scientists without understanding physics involved. Be humble. People raising AGI X-risk arguments are smarter than you and gave it a lot of thought. Put an honest efforts in understanding their arguments.

Bostrom is convincing if you treat his postulates as axiomatic, the same way he does. I don't. Do you think rereading an unfalsifiable argument will make it less unfalsifiable? Or that a naked appeal to authority counts for anything at all, except in the realm of religious faith?

Imagine the real world friction overcome to evolve a human from a single celled organism.

Now consider that a computer program can replicate and change itself orders of magnitude faster than a human generation.

I've dealt with spaghetti systems that need such maintenance and I'm unconvinced that poor devops is a major barrier to the hypothetical of runaway AI. Devops people will probably be using AI to fix these issues soon enough if they aren't already.

I agree that a lot of the AI risk pundits are either people with no real AI experience (or very outdated, resting-on-their-laurels experience) or if they do know AI, they probably know little about the complexities of running actual large scale, business or mission critical systems.

However, in their defense, the real question is whether those types of problems are something this new class of AI could potentially excel at. I've been an AI skeptic for decades, having had real world industry experience trying to get it to solve problems, and this is the first technological advance that has actually surprised me with its capabilities.

AI Risk still doesn't rank above many current, pressing global problems for me personally, but it also doesn't seem ridiculous anymore.

One of the things that keeps me rooted is that I've worked as then employed teams of experienced DBAs to maintain "self healing databases" for last 20 years. Large but non crazy / non special databases with robust reliable technology that just need tender loving care all the time. And I'm increasingly aware as I age that I am not special and everywhere else is not that much better. So I'm with you on the dangers of self running / self improving stuff running away, vs 10k immediate risks and issues (as in, happening, not hypotheticals).

Hook up the model's output to a shell. Remember it doesn't have to be engineered to modern SRE standards, it just needs to not crash. Dirty hacks are on the table. The comparison to diaper-changing is fairly illuminating, actually: even human children, let alone adults, figure out how to take care of that kind of thing, and it fades into the background of their activities. I agree that plausible mechanism is lacking in these AI risk conversations, but ops is not the hard part IMO.

A first stop could be that the LLM becomes good enough at economic manipulation to pay humans to maintain and upgrade itself.

A possible next stop is that the LLM becomes better than humans at designing new algorithms and hardware.