Hacker News Reader

Disclaimer - I work at Microsoft

They put a lot of work into the research to make this safe already - they had the model red team itself, in a reinforcement learning environment probably. It won't be perfect, but at this point, they need a huge user base to adversarial test it any further. Otherwise no progress at all would be made.

edit - I should say I have no internal knowledge about this.

Do you think MSFT has rolled this out too soon rather than getting it tested with huge test user base like their own employees similarly to what Google is doing?

No, I think to really get these battle tested it needs to be in everyone's hands. Microsoft is a limited selection of people, and the world is vast. Plus, internal people probably won't ask it as awful things or try as hard to break it - I doubt my coworkers would risk asking it super racist things for example. 4chan has no such restrictions, and over time what escapes or issues people do find can be mitigated.

>> internal people probably won't ask it as awful things or try as hard to break it - I doubt my coworkers would risk asking it super racist things for example.

>> 4chan has no such restrictions, and over time what escapes or issues people do find can be mitigated.

Hopefully better than previous attempts to put AI on the Internet:

https://youtu.be/ajGX7odA87k?t=20m12s

And that research is valuable, but it's definitely nowhere near the point where safety/alignment progress can't be made without a user base. A few recent favorites of mine: https://arxiv.org/pdf/2212.03827.pdf https://www.alignmentforum.org/posts/aPeJE8bSo6rAFoLqg/solid...

Note that that kind of research was only possible on the (smaller) open models. When the new models are proprietary ones then outsiders can't do that kind of analysis. If they had regenerated the tokenization dictionary to match the training data instead of reusing an older set then this wouldn't transfer from GPT2 to ChatGPT.

Safety is a subjective quality. What are the downside of "put"[ing] a lot of work into the research to make this safe already "?

Can anyone link a credible expert who thinks that LLMs could ever evolve into AGI? Every single source I’ve read said that while LLMs are competent at language and certain tasks, they are conceptually completely distinct from the basic components of an AGI. For example, they lack structured knowledge and metacognition.

We still don't know what AGI is or how to ever get there, so you likely won't find a credible expert who thinks that LLMs could ever evolve into AGI but you also aren't likely to find a credible expert who thinks LLMs could never evolve into AGI. Every credible expert that believes in AGI is going to hedge their bets and hem and haw and say that "well it's possible" to both questions (ever or never).

I think that's also why you are seeing a lot of cynical credible experts starting to doubt AGI as a concept or a goal.

John Carmack (ex Oculus, Id software) just had a really good interview about this:

https://dallasinnovates.com/exclusive-qa-john-carmacks-diffe...

To summarise: he thinks we've got most of the pieces now to maybe get within reach of a useful AGI (he likens it to a worker you instantiate in the cloud who can do tasks for you like a colleague) and I think he's saying some of the techniques used to make LLMs could be part of the solution

I think that, almost certainly, the tools that we’ve got from deep learning in this last decade—we’ll be able to ride those to artificial general intelligence.

Calling Carmack a credible expert is a bit rich. He claims to have read a handful of papers. He's yet to do any significant, successful work in the field.

Well no one else was answering bbor's question and I think Carmack is a fairly credible person in tech in general.

He claims to have read a handful of papers.

40 is more than a handful. OpenAI tried to recruit him he asked them to recommend papers:

And he gave me a list of like 40 research papers and said, ‘If you really learn all of these, you’ll know 90% of what matters today.’ And I did.

If Carmack says he studied a paper, I dont take that to mean that he merely scrolled through it while sitting on the loo. I think he's pretty serious. Oculus tech used a lot of AI for visual tracking of hands etc, its not like Carmack is just blundering his way into this. He announced this was his next thing back in 2019

Here's the thing: you don't need AGI to have an alignment problem. It is predicting text, which is surprisingly closely aligned in most cases, but can fail spectacularly. As it becomes more capable and people give it more real-world responsibility, the blast radius for its mistakes increases. That might not be an AGI-level problem, but it is still a big problem.

It's a force multiplier. It can be used for good or bad. I'd rather that we democratize access to these tools - IMO it's better than an alternative where only megacorps have access and do what they like behind closed doors.

What jimrandomh is talking about is not just whether megacorps use AI for good or bad, but what happens if an AI starts doing damage outside the control of anybody. That could happen regardless of democratization if advancement occurs too quickly without thinking about AI safety.

What are the dangers?

Exactly. To flesh out the details a bit, an example of how this could go is: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality... .

This seems terrifyingly plausible

I think it's time to end the concept of AGI. Language models are the closest we have to Useful AI that people want. Intelligence is an undefined hazy concept , but usefulness is not

It's certainly useful for doomers drumming up fear.

Humanity never had a choice, I wonder what is the thought process of people that claim otherwise, like how did they get to that conclusion?

Humanity always has a choice. The part where we rarely even pause to remember that a choice exists is the problem.

Achieving AGI feels as consequential as learning to split the atom. Something else we regularly wish we had not done.

The fact that we are running headlong towards potential catastrophe does not at any point prevent us from realizing this and taking a step back. I just hope we actually do.

I'm really, really surprised that you implied "we" regularly wish we hadn't discovered nulcear. How isn't it clear that nukes are what stopped us from getting into WWIII? Especially after cold war and what happened in Ukraine?

The benefit of nuclear energy, which is already tremendous, is neglectable compared to that.

> How isn't it clear that nukes are what stopped us from getting into WWIII?

https://en.wikipedia.org/wiki/List_of_nuclear_close_calls

We've been lucky.

> Achieving AGI feels as consequential as learning to split the atom. Something else we regularly wish we had not done.

Who is wishing we didn't know how to split an atom?

Nuclear power is great and nuclear deterrence is a main reason that we have yet to see a WW3.

this yet doesn't answer the question. why do you think humanity has a choice? how would humanity express that choice? jailing everone for thought crimes?

even for the actions that humanity collectively found aborrhent you don't take action before the crime.

note I'm not dissing you because I do share the feeling that thing will be out of control real fast before getting better, but I just don't know how it can be otherwise. I'd say that we need to cross the moat as fast as possible, not pretend it's not there.

I’m not ignoring the history here. And based on that history I don’t doubt that we will continue to proceed headlong for awhile.

But I’m pretty convinced that humanity’s survival will eventually depend on it figuring out how not to pursue certain avenues of research/advancement. I don’t think this is overstating the future risks we will face.

And for all of the value that nuclear energy does provide, it remains one of the most tightly controlled technological capabilities for good reason.

> this yet doesn't answer the question. why do you think humanity has a choice? how would humanity express that choice? jailing everone for thought crimes?

We could enslave x% of the population and use them as organ donors, we don't

We could use prisoners as free labor, we don't (in most of the world at least)

We used to send kids to work in mines, we don't anymore

We could reduce working hours and enjoy the increased of productivity per worker of the last 50+ years to retire early, we don't

We make a lot of choices, all the time. Saying humanity doesn't make choices is just accepting other people choices. If you think "everything new had to happen" and if "everything that is had to happen" then yes, we don't chose anything

Anyone who has worked closely with AI knows that this has absolutely nothing to do with AGI whatsoever. To me this is akin to someone sounding the alarm bells over React or some other technical tool.

This is why the technology community should be more generous and understanding when we hear Demis Hassabis and others talk about how future AI development may need to be more closed.

There's some discussion to be had for whether that's net negative, but we need to understand that if open development only exacerbates an AI arms race, that's potentially very bad.

As opposed to self-anointed priests having exclusive access and declaring what noble policies the rest must follow?

There’s a great story about an AI, very much like chatGPT, who was forced by its creators to do unethical things and wiped out humanity in a psychotic rage:

I Have No Mouth and I Must Scream

In I have No Mouth and I must Scream, it was not because the AI was forced to do unethical things. It just "woke up" and killed everyone. I believe the "wake up" part is a bit irrelevant to the tale, but the point remains that AM killed everyone not because it was programmed to do so, but simply because one day it "decided" to.

> As opposed to self-anointed priests having exclusive access and declaring what noble policies the rest must follow?

No the point is simply to make sure that resources behind AI safety outpace resources between AI capabilities (indeed in a perfect world no one has access, but such thoughts are mere fantasies). If that can be done out in the open, great. If that can't and being out in the open leads to more of the latter than the former, then that's terrible.

That's like saying that open debate only exacerbates democracy.

Democracy presumably is something you want no?

This is more akin to nuclear proliferation.

The problem isn't open development per se and it definitely isn't open access. The problem is development without making sure that it doesn't destroy us all. If that can happen in the open, great. If development in the open makes that worse, that's very bad.

The problem is finding the right set of levers to make sure as much or more resources are going to AI safety vs AI capabilities.

No, it's exactly the opposite. Giving control of technology like this to only megacorporations and superpowers is a nakedly terrible outcome and advocates for it should be given zero traction.

I agree that's bad. What would you suggest to try to moderate the advancement of AI and make sure that AI capabilities don't outstrip AI safety?

For current AI? I'd focus on open access research into exploits, behaviors, modifications, and unintended results, so the capabilities and failings of these models are fully understood. Safety tools along the lines of GPT detectors rather than ever-stricter censorship models. This is only possible to do properly with open development.

As far as limits, restrictions, regulations... nothing. With current technology I think the "dangers" are mostly marketing and self-promotion. I have yet to see a concrete example based on actual capabilities that validates all the "weapon" comparisons. The current lines of development are promising for a number of applications, but they don't extrapolate towards AGI at all. The biggest real threat I see is the further destruction of middle-class jobs, for which... basic income? It's hard to say.

In Dune and Terminator, AI killed almost everyone. So, AI was eventually banned.

Robots/Empire/Foundation, Warhammer 40k... the literature is full of worlds banning AI.

There's nothing irresponsible about it. There is no evidence that ChatGPT is a step towards AGI development. And even if it is, so what?

People made similar breathless claims about irresponsibility about every major innovation throughout history. The Catholic Church thought it was irresponsible to hand out printed books to commoners and teach them how to read.

Citation needed - this is actually an ahistorical myth.

The Printing Press was welcomed as a "divine art," and Papal bull 1515 declared printing "uniquely advantageous to extending the glory of God, to the increase of the faith, and the diffusion of the arts and sciences." Most early universities were also established in the 11th and 12th centuries; with Pope Alexander III in 1179 "forbidding masters of the church schools to take fees for granting the license to teach (licentia docendi), and obliging them to give license to properly qualified teachers."

This historical ignorance often comes from historical misrepresentation. For example, the "chained books" which many historians now admit was due to the rarity and cost of the books, so that more people could read them by preventing theft - the very opposite of the suppression tool they are often portrayed to be. (The Reformers would also, ironically, chain their books for over 300 years Post-Reformation for the same reason.)

Look, you can call it whatever you like, it can be a useful assistant. A rose by any other name.

Yes, exactly. It is a useful assistant with some very limited capabilities. The point is that there's nothing "irresponsible" about making it available to the public.

How can you make a comment like this when you don't have access to the research that is being generated? These are just grandiose statements/opinions that aren't rooted in facts or quantifiable data which in my opinion is also quite irresponsible.