Comment by cmpb | Hacker News Reader

Comment by cmpb | Hacker News Reader

On their "OpenAI Charter", they list several basic principles they'll use to achieve the goal of safe AGI, including this one, which I find pretty interesting:

>We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”

If I'm reading that correctly, it means that later on if/when some company is obviously on the cusp of AGI, OpenAI will drop what they're doing and start helping this other company so that there isn't a haphazard race to being the first one, which could result in unsafe AGI. That sounds like a well-intentioned idea that could cause more problems in practice. For instance, if there are multiple companies with almost equal footing, then combining forces with one of them would give a sense of even stricter deadline to the other ones, possibly making the development even less safe.

Also, they only mention assisting "value-aligned, safety-conscious" projects, which seems pretty vague. Just seems like they should give (and perhaps have given) more thought into that principle.

We had in mind less a mechanical trigger ("project X is within Y years of AGI, let's drop everything and join them"), and more a broad commitment to avoid races, along with an invitation to other organizations to build relationships focused on ensuring a good outcome. In practice we plan to be in constant communication with other major AI orgs about these issues (in some cases we already are), and eventually we might hope to help build multilateral agreements that would avoid the kind of coordination issues you describe. This will be an ongoing process, playing out over years, with lots of details that need to be worked out. The charter simply announces our commitment to see this process through.

On "value-aligned, safety conscious" projects, we wrestled a lot with this wording, but we believe it's the best way to describe our important caveats. There has to be some level of malicious use at which we wouldn't be okay cooperating with a project. And there has to be some level of neglecting safety considerations that would also make it unethical to cooperate. Our message here is that aside from these caveats, avoiding a race is the most important thing. In practice we expect (hope?) that will be many value-aligned, safety-conscious organizations, and again the conversation around these topics will play out over years rather than just being a random decision we make.

More generally, on both points our intention was to make a broad statement of values and intent, rather than to nail down precisely what actions OpenAI will take. The central document of an organization needs to be both short and flexible enough to remain relevant for many years, and that necessarily means sketching a broad framework and leaving the details to be filled in later. That said, you should expect us to fill in many of these details over time, both in explicit documents and in our actions. In fact, we are building a policy team that is focused on these issues, and it's hiring: https://jobs.lever.co/openai/638c06a8-4058-4c3d-9aef-6ee0528...

The problem is they're trying to filter the production of a group of unrelated organizations, without any recognized authority to do so. The vagueness of the principle reflects the intractability of the goal, in the current environment.

It's not even clear how to evaluate whether anyone "comes close to building AGI".

Have they defined what "AGI" is supposed to be? I can't find it on their website.

Reply on news.ycombinator.com