Hacker News Reader

Comment by kmeisthax | Hacker News Reader

I think what's already obvious is that most people in AI research care only for solving some hard problems in computer science. Both AI safety and ethics[0] be damned. So the amount of control OpenAI had on the research arm is less than you'd think.

[0] For the record, AI safety is the branch of research worried about controlling computer God, while AI ethics is the branch of research worried about whether or not it's OK to pay Pakistani kids pennies to label training data.

View on HN

I'm not sure that's accurate. Most companies use "AI safety" as a shorthand for "brand safety", i.e., making sure that your LLM can't be trolled into praising Hitler or denying climate change. Basically, do what needs to be done to avoid bad press.

There is definitely that second and more abstract take - AI doomsday safety - that Altman liked to talk about to get favorable regulation passed. But it doesn't seem to be much of a real field. "Hard" AI doomers think that superhuman AI can't be reliably constrained and we should just stop. "Soft" doomers think we need to solve "alignment". But we don't really know what's going on inside an LLM, so we test for alignment by asking it what it thinks about Hitler. In other words, back to brand safety.

My company Preamble markets ourselves as an AI Safety company so I’ll opine on what I think is a reasonable middle-ground definition. We interpret near-term AI safety to mean solving the problems that already exist today but will also exist in AGI.

As a key example, we were the first team to discover Prompt Injection, which we responsibly disclosed to OpenAI leadership in early May of 2022, four months before it was independently rediscovered by Riley Goodside.

(See the citation at the bottom of this page https://github.com/OWASP/www-project-top-10-for-large-langua... )

Prompt Injection is a perfect example of what we consider a near-term “AI safety” issue: it’s a real issue that affects current systems, and it will also affect any instruction-based AGI systems if a solution is not found by the time AGI is reached. There is currently no known general solution to Prompt Injection, and it’s #1 on OWASP’s recent list of the top vulns in LLMs. (See link above. This vuln is designated LLM01.)

You're not wrong. AI safety originally meant the doomsday variant and has sort of morphed into the brand safety variant. My understanding was that safety research was treating both as the same problem, or at least that they believed brand safety was a worthwhile subset of the doomsday problem to attack.

I thought AI ethics was not creating racist bail predictors or health insurance bots that automatically deny claims with a 90% error rate.

It's a broad field.

Also, the two are tied: those underpaid, overworked Filipinas? Surprisingly reactionaries.

Genuinely, great distinction.

Would you also include in “AI ethics” a considered rollout of automation to maximize its benefit to everyone? Because there is real benefit. The kind of work AI can automate is shit work and it would be to the theoretical benefit of society for it to be automated… unless all the benefits accrued to the owners of the tech instead of the people whose (stupid) jobs are being replaced.

Is that “AI ethics” or still a third (anti capitalist?) thing?

> unless all the benefits accrued to the owners of the tech instead of the people whose (stupid) jobs are being replaced.

The workers doing the jobs will never benefit. They'll be shown the door and the owners will reap all the profits.

Reply on news.ycombinator.com