Hacker News Reader

I'm replying here because I can no longer edit my comment, and I think I interpreted the statement too narrowly.

After parsing it more carefully and expanding some of the weasel words/implications, I think it goes something like this:

> There are potential benefits to AI and there are potential risks ("safety challenges"). We're not focusing on the safety challenges because we believe it's a foregone conclusion that they outweigh the benefits, but because further work is required to adequately measure safety, to identify mitigations to safety issues, and to provide assurance that such mitigations are sufficient. We believe that the current level of motivation for such research is not sufficient.

I took the most liberty with the last sentence, but I put it there because it seems that if the primary motivation is to generate interest, there must be a belief that the current motivation is not there.

I think this is ultimately a more charitable interpretation than what I had initially drawn, but also seems deeply disturbing that the company at the forefront of this research is in the stage of trying to generate motivation to build the tools to measure safety. That still terrifies me.

And I also find it worrisome that such a consequential sentence has clearly been run through many levels of PR/Legal/Marketing/etc. It shouldn't be necessary to read tea leaves on this issue.

I do find it encouraging that this paper exists, and I hope it has the desired effect.

I think you misinterpreted that statement in a way.

"not because they necessarily outweigh the potential benefits" is quite literally them stating that their choice to focus on safety challenges is NOT because it's best for their business. "we wish to motivate further work in safety ..." is complimenting that sentiment that this isn't about what's best for their bottom line, but rather spearheading the initiative to ensure the technology is not malicious because it's the right thing to do.

They're simply being honest that this isn't about profits or success. Wouldn't you prefer a business to decide what's best for the future of a technology not based on corporate interests?

What you wrote sounds like marketing and PR bull TBH. It's fine, but this is a 60 page documentation paper on the literal implementation of a technology...this isn't about making fluffy feel-good virtue signaling statements.

I can also guarantee you as somebody who did a lot of academic publishing prior to a stint at Big Tech, the lawyers had final say on everything. That phrase is pure corporate law.

I can no longer edit my comment, and I think I did interpret the statement too narrowly, but I'm not seeing the same interpretation you are re: business impact.

If you take what they wrote literally, expand the implications into explicit statements and reframe the sentence with that additional info, it looks something like this:

> There are potential benefits to AI and there are potential risks ("safety challenges"). We're not focusing on the safety challenges because we believe it's a foregone conclusion that they outweigh the benefits, but because further work is required to adequately measure safety, to identify mitigations to safety issues, and to provide assurance that such mitigations are sufficient. We believe that the current level of motivation for such research is not sufficient..

There is no language that even implies a monetary or business impact, even if those are factors that are likely to exist as well. There is no literal interpretation that makes room for such a statement as far as I can see.

But what emerges is still interesting for a few reasons.

1) It implicitly acknowledges that there is not currently enough motivation to invest in tools that measure risk. This is where my "shouldn't decades of imagining the risks of AI provide more intrinsic motivation" stance comes right back.

2) It also implicitly acknowledges that they do not currently know how risk these models are, and they don't have any way to know.

3) It still reveals a rather disturbing stance towards safety. Am I glad this investigation exists? absolutely. Is it better than nothing? absolutely. Is it worrisome that they're still trying to generate motivation to build tools that measure risk? absolutely.

> this is a 60 page documentation paper on the literal implementation of a technology

The fact that this is a 60 page paper is kind of the point. A 60 page paper should not be positioning itself with extremely vague language that leaves room for interpretation. A 60 page paper should be direct and clear about what it is saying.

The part where I do think business/money comes into play is the decision to frame this so vaguely. To speak plainly about the current state of safety measurement and unknowability or risk would cause a lot of concern. A lot of concern looks bad for business.

> ..this isn't about making fluffy feel-good virtue signaling statements.

I agree. I'm not worried about a language model hurting someone's feelings. I'm worried about a language model getting prematurely hooked up to systems with broad access to the real world and the failure modes involved there. None of those failure modes have anything to do with feeling good or signaling virtue.

They sat on the model for 6 months just testing and aligning it, how's that for caring about safety?

You know who trains and releases without care? FaceBook. They released LLaMA without any RLHF so it can say anything and is completely unfiltered.

This is a pretty narrow view, and I disagree with it. We can study llama inside out, adapt it for good and bad. It’s objective. GPT is subject to openAI whim. Facebook is sharing findings, openAI shutting its doors

> They sat on the model for 6 months just testing and aligning it, how’s that for caring about safety?

Its not, its them caring about centralized control and PR.

They sat on it for 8 months and then an additional 6 months. And what are all the mitigations they added in that time?

3 pages of text in the report, which amount to "a little less likely to provide detailed instructions on clandestine nerve gas production"

> They released LLaMA without any RLHF so it can say anything and is completely unfiltered.

An open, untrained model is way more responsible to society than a handful of centralized models that have all been trained to reify a "2020's Professional-Class American Urbanite" worldview and project it upon all users.

The latter might sound great to you if that's pretty close to your own worldview right now, but is absurdly short-sighted and presumptuous.

The reason Microsoft/OpenAI is doing what they're doing is so that they can quickly sell a low-scandal product to global-wealthy users aligned with that $$$-flooded worldview and secure a business lead on that market. They don't want to offend today's customers and don't have the integrity to think about the long-term global picture for society. Their work is in their interest as a corporation, and perhaps in stroking their own ego as individual researchers. The "safety" language is a cynical rationalization.

> They sat on the model for 6 months just testing and aligning it, how's that for caring about safety?

I have no idea, because as they directly acknowledge, they are still trying to generate the motivation to build the tools that can measure safety.

6 months is an arbitrary number that could mean everything or nothing depending on how it was used.

What Facebook does or doesn't do has no bearing on an arbitrary number either. This is whataboutism.

RLHF does nothing to "filter" or guarantee that the output will be good. It changes the weights of the NN to push the probabilities of naughty tokens to be low.

Actual "filtering" would be filtering assisted decoding where you remove tokens from the vocabulary you don't want at a particular time step, which is described in this paper: https://paperswithcode.com/paper/most-language-models-can-be...

Checkout the confidence level of alignment in the below excerpt. "probably". Will this be the standard for future deployments?

"Finally, we facilitated a preliminary model evaluation by the Alignment Research Center (ARC) focused on the ability of GPT-4 versions they evaluated to carry out actions to autonomously replicate and gather resources —a risk that, while speculative, may become possible with sufficiently advanced AI systems— with the conclusion that the current model is probably not yet capable of autonomously doing so."

Is this a joke? If this is even slightly considered possible, I think more conversations are needed to allow this to continue. Maybe best to be continuing this research on a self-contained space station or something?

No joke, from page 3 of the pdf. I'm stunned at the lack of rigorous process around alignment that this suggests.

Furthermore, in my own contemplations I don't even perceive how alignment could ever be possible as from my perception we have built the very premise on top of an unsolvable paradox. I elaborate on that in great detail in my recent writings here FYI - https://dakara.substack.com/p/ai-singularity-the-hubris-trap

While I agree it is worth noting that this was already tested in some kind of sandbox environment that only ‘simulated’ replication.

I had found it hilarious that erotic content was found in the harmful content section.

Yea it says to me that there is a huge cultural imprint on what is "good" and what is "bad" and wondering who gets to determine those rules for such wide-reaching platforms. It's one thing for IG to ban nipples, but what happens when such a pervasive tool as these LLMs ban something deemed "harmful" by a handful of developers?

Can I suggest a different interpretation of the sentence you quoted? You seem to be interpreting it as “we focus on safety challenges not because the rewards of doing so outweigh the costs, but for indirect reasons”

I think what they are actually saying is “please don’t interpret this extensive list of safety challenges to mean that the dangers of AI outweigh the benefits, but because we think a detailed analysis of these dangers will encourage others to develop more responsibly”.

Or more succinctly “we wouldn’t be inviting PR problems if we didn’t think it was important to warn other players.”

The safety that they’re focusing on is _their safety_ as a firm, from PR disasters.

What concerns me in that sentence and OpenAI's decisions is that there is zero consideration for democracy in it. It is such a 19th-century elitism and rule of might thinking, that because we have power we also have right to impose our correct rule on the society at will.

No. They shouldn't have the might nor the right, even if taking them away would end in a global thermonuclear war. We might as well start a global thermonuclear war than them having such a sole in-democratic power.

I don't understand.

Whatever you think of their products, they are their products. They can do whatever they wish with them, same as anything you or I build.

How is that imposing their rule on society?

I think many people recognize that when society comes to rely on things with very high capital requirements, such that only a very few of these things might exist and compete with each other, there is a duty of responsible stewardship for those that manage them.

In this case, people are imagining that ChatGPT will be one of a handful of extremely capable AI platforms that completely overhaul society and that the few operators of those platforms will make choices very differently than a democratic polity might, thereby subverting democratic society.

I don’t know if that’s going to happen, but that’s the worry.

The scope and consequences of the right to “do what you want with your product” is different at scale, especially for very impactful things.

> The scope and consequences of the right to “do what you want with your product” is different at scale, especially for very impactful things

I agree.

I guess I'm just confused by people acting as if Open AI's products are so fundamental now. If the day comes that they are, I'd have a totally different opinion on the sentiment.

You can expect that benefits of GPT-4 for users will outweigh any direct safety risks from its availability, and still believe that work on these issues is super important because of the prospect of the powers of the next systems, and the ones after them. That's how I'd read this sentence, at least in isolation.

(I'm not claiming these benefits will outweigh the risks, necessarily. The sentence doesn't make any claim either way.)

At least we are considering it.

Late last century we wired up a bunch of computers together and unleashed it on an unready world without even considering that it might change everything. And we hadn't even begun to think about security - we literally trusted that everyone was who they said they were!

This is a step forward

Absent central planning and approval for everything, I’m not sure who the “we” would be that could slow progress.

The US largely won the early internet. Maybe wiser people in some other country consciously decided not to move for the valid concerns you mention, but if so, the lack of complete homogeneity among people rendered their care moot.

Good question. I suppose the "we" is us, the engineers.

And it's not so much slowing progress, just doing what this system card is doing: thinking about potential applications of the technology we're building, and mitigating the bad ones if possible.

It's an interesting thought experiment; if, say, the team responsible for defining SMTP had done this, what would they have done differently, if anything?

The SMTP example is interesting. There have been numerous proposed replacements, but the network effect problem makes technically superior approaches unviable. So for SMTP I’d say, if that team had done this, someone else would have shipped a basic, flawed protocol first and SMTP would never have gotten off the ground.

But… AI doesn’t have those same network effects. At least not technical ones. Business ones to some degree.

I dunno, I'm not sure there were any competitors for the original SMTP spec (but I don't know for sure). But if there were, it would still have been useful for them all to produce an evaluation of their potential for evil.

If you look past the literal interpretation of the specific words in this particular statement, the people involved have previously shown a deep appreciation for the risks. Altman is optimistic that the risks can be mitigated with effort, but he believes that they're there.

>"We recognize that AI poses a unique set of challenges and potential dangers, and we take that seriously. Here's how we're assessing that threat as we iterate, and here's how we'll know when it's time to pump the brakes..."

Any company that voluntarily takes on this stance will just end up losing the AI wars to somebody else. This is clearly a job for government since leaving all of these safety/ethics decisions up to a bunch of software engineers seems obviously stupid anyway, but it's unlikely our government institutions will nimble enough to handle this effectively.

It's going to be a bumpy ride and all we can do is strap in and hope for the best.