OpenAI’s head of ethics leaves less than a year after joining | Hacker News Reader

OpenAI’s head of ethics leaves less than a year after joining

ft.com

524 pointsilamont489 comments

https://aimagazine.com/news/why-did-openai-head-of-ethics-ch...

Organisationally it never works to have a group whose only job is to say no to some other group. The incentives are diametrically opposed and as a structure it can’t last.

If you had an AI company and want it to be ethical you have to find a way to make ethics everyone’s responsibility, and have the consequences of poor ethics bite the people who make those bad decisions. If you just outsource it to the ethics group what happens is

1)everyone else thinks they don’t need to worry about ethics

2)the ethics group need to justify their existence so introduce a bunch of guidelines that everyone initially thinks are reasonable but over time people think are increasingly out of touch

3) The ethics group start to “make difficult calls” and say no to things. Initially everyone supports this and feels like the system is working as it should but over time everyone starts to just see them as an obstacle to work around

4)everyone else starts to try to work around what the ethics group says

5)The ethics group grows powerless and disconnected. The people who work around them “get things done” so get promoted etc whereas they only visibly put roadblocks in peoples’ way, so they get sidelined.

6)Eventually they get disbanded with some corporate announcement thanking them for their hard work, thought leadership etc. All that has been achieved is a lot of wasted time and bad blood.

If this were true, food companies wouldn’t have a Regulatory team. Except every single one of them does, and they function similarly to how you describe, but without dissolving. They’re treated as a constraint that must be checked before major projects can advance. Sometimes they’re the longest-lead item on a new project and if you don’t get them involved early, your project can sink at a late stage after great expense.

The only difference is AI isn’t regulated, so they just have a vague “ethics” department with no real teeth because it’s essentially PR and has no legal consequences to back up their stance.

> The only difference is AI isn’t regulated

Seems extremely obvious that this is exactly why the food industry doesn’t dissolve its regulatory team?

I don't think that's a fair comparison. Regulatory teams exist to make sure companies comply with actual laws and regulations. There's a relatively objective standard they're enforcing.

Big tech has the same thing in its legal and compliance teams, and those teams absolutely can kill projects. That's pretty different from an amorphous "ethics" committee deciding what is or isn't ethical based on much more subjective criteria.

ethics isn't regulation. regulation is external, forced upon them, and has direct usually 1-to-1 consequences if you don't comply.

ethics may feed into regulation, hopefully, but the regs may just as easily give the orgs they regulate a pass on things they shouldn't; e.g. "regulatory capture".

Food companies aren't rapidly developing groundbreaking new technologies on the most competitive battleground in computing history where everyone is trying to compete globally just to create the foundational infrastructure.

It's not that hard to get some pickled eggs on a grocery store shelf while remaining compliant with expected ethical standards for the industry. Food is a pretty mature industry.

> rapidly developing

if the speed comes at the cost of not making reasonable ethical considerations, is that not something that should be looked at?

Regulatory teams are about external compliance. That’s different from “chief ethicist.”

It’s a little unclear what this person was doing. But ensuring compliance with government regulations was probably not their core function.

I don’t view that team as saying “No”.

Instead, I view that team as saying “Customers expect safe food. Our job is to help you deliver on that”.

That means they’ve learned how to say “no” more diplomatically :)

> The only difference is AI isn’t regulated

You seem to believe that regulation is "ethics backed by law".

As this is not true, your counter example is not even on point.

Food companies produce literal poison. Not a great example

Just think what they'd produce without regulations.

Poison can act on more than just your stomach

I get what you are saying but that’s exactly how security and compliance work. And I don’t think amount of people who want to be ethical in an organization ia that different than people who want to build secure software (but possibly no one gives a shit compliance other than it’s something that needs to be done)

And organisations fail at security when security and compliance is only considered important by that one teamnn

Security requires the whole business to buy in. And it requires processes that allow people to get shit done without people resorting to shadow IT; thus working around that one team.

So the GPs point still works.

The difference between "ethics" and "security and compliance" is that the latter is something that hits inside the company, while the former usually hits outside of the company.

Poor security practices harm your teams, your data, and usually you make moderate savings at best. Poor ethics "only" harm your customers while making bank for the company.

This is the real problem with ethics in a large corporation. You're not saying "no" to another team, you're saying no to large profits, you're saying no to the company's leadership. That is what never works.

Rule number one of CIO: become friends with CFO and chief lawyer. And nothing else matters.

In a high-functioning security and compliance team they tend to say “no” only in really dire circumstances. Good security and compliance teams spend a lot of time asking exactly what it is the people are really trying to do and then find a way to say “yes, and…” as in “yes you can and here’s how you do it without compromising security/breaking the law etc”. And as a sibling said, orgs fail at security when they make security only the infosec teams’ job.

This is also why I said “whose only job”. In a good org, the security team doesn’t only say no to devsecops requests, they also do trainings to skill up other teams, keep the network secure, proactively seek out and understand external threats, work with external vendors etc etc …

I think the key part is about aligning incentives and outcomes - if your security and compliance departments are only judged on "were we breached, did we pass our audits" then there's a risk they won't weigh the tradeoffs associated with the decisions around controls implemented to achieve those outcomes.

To use a ridiculous extreme you can't breach a web app that isn't exposed to the internet, but the users can't access it either.

If you can connect/balance those goals to other metrics around cost and productivity, usability, and a realistic threat model, as guardrails then you incentatize cross-team collaboration to achieve the shared org outcomes.

The structure is untenable when the work of one team is to say "no" to other teams, when these teams are not asking.

A good infrastructure team would seek a competent security review that would say "no" to problematic things before an intruder says "aha" to them. If feedback from the ethics team is not sought, nobody is going to heed its opinion anyway.

I’ve never known a security team to be in the position to say “no.” The role of security teams vary a lot from one organization to another.

But generally they can make you aware of the tradeoffs you are making, or serve as a kind of quality control that generally does not want to be asked questions.

Security/compliance have the external hammer: if they fail, your org will have to pay fines and someone may end up being criminally liable (depends on country).

The ethics department; if they fail, there may be some negative journalism, but who which AI company has positive journalism these days? There's no external hammer for ethics.

SRE is a 'say no to powerful people' job as well. I think for this to work the leadership needs to show support for it. The friction is still there, but in more tolerable areas. I bet for ethics this isn't the case at OAI, but everyone values security and stability.

For an SRE there can be more directed hate received from the junior employees, that want to release new features they developed. Especially because there is less accountability across orgs. Security is an interesting one because it seems to have less of this friction, maybe because it's more clear cut what is an issue.

During my years at big corp we did tons of work just to go around the law department. While for core business it made absolute sense to have lawyers involved in daily business we were satellite office decoupled from core business. Local management failed to communicate that and we were stalked by lawyers from main office. Pretty sure others do the same with compliance, law, ethics departments. Just work around to get things done instead of stuck for weeks in stupid meetings.

Markets solve this by pricing risk. Ideally you would have some kind of notion of selling insurance internally and track things. But ultimately existential risk is hard to negotiate from the inside. And companies are supposed to go bust or succeed. They are not really the same as a population trying to survive forever. At least that is one take.

Security and compliance tends to be a lot less subjective than ethics. Not saying it’s objective, but there’s a huge difference.

The downside is it can often feel like a box-checking exercise than actual security or compliance, but “you need 2FA” is less debatable than, say, AI and copyright.

Big corporations there is politics on play and too often you see CABs and other bureaucratic stuff instead of checkboxes for gated releases, 2FA etc.

The compliance team seeks to maximize company profit/success within the confines of hard legal boundaries.

There are no "hard" boundaries for ethics or "doing the right thing," and so those boundaries will always be pushed until they are useless.

> exactly how security and compliance

Not everywhere. I go out of my way to assist teams to achieve a secure outcome with less effort.

Things like: “instead of admin access to the production servers the devs can have fully automated deployment pipelines combined with OpenTelemetry for observability so they don’t have to spend half the day scrolling through gigabytes of logs.”

That’s more secure and and more better.

Nobody had to be told “no”.

Similarly, I replace key store access with secret-less managed identity, etc.

There are at least some penalties for failed security and compliance. Ethics is the first victim of greed.

In a situation where it doesn’t happen this way, the ethics team grows in power and staffing over time until it destroys the company, because ultimately all commercial activity can be construed as less than perfectly ethical at some level, or it morphs into an elaborate legalese department that masks unethical practices in a cloak of justification.

Either way, you need third party regulation and then a department dedicated to ethical compliance, or you end up with corporate cancer.

I say this as a person with a knee-jerk anti-regulation reflex.

> whose only job is to say no to some other group

I suspect that people will miss just how precise you've been there.

Your proposal - if I understand correctly - is not just to spread the "ethics team" onto everyone (which is correct), but also to have someone in a dedicated role. However, they need to have not _just_ that role, but also some skin in the game.

This is also my view. These sorts of roles can have a very high impact and be very successful if everyone feels like they're rowing in the same direction.

The people in those roles need make people feel like there is value in seeking their input. They also need to find ways to say yes that helps things happen in the right way, rather than stopping them entirely.

This happens naturally in organisations where security is valued by everyone. InfoSec / CIO roles can operate very successfully and deliver a lot of value. As soon as people lose faith and see them as the "no" team, they start trying to hide from them.

There are lots of ways to create this but it's as much about the people in the role, as it about the organisation itself. If either are skeptical about the other, it falls apart very quickly.

Operations was like this so we added Ops to the Dev. DevOps.

Security was like this so we added Sec to the DevOps. DevSecOps.

Ethics is now the problem, so we’re going to add Eth to the chain. DevEthSecOps.

Legal and compliance is now the final hurdle to shipping fast. AI tools will allow us to implement DevEthSecLawOps at scale no doubt.

Absolutely - though the flip side of this is if something is everybody responsibility - it's nobodies.

There are two things to successful organisations structure and people, you are focusing only on the structure.

What you need for somebody responsible for health and safety, ethics, security, or compliance is a person who has courage and is willing to take calculated risks.

However these roles do have a tendency to attract the risk adverse, or make them risk adverse if you punish risks that go wrong too harshly.

And a key critical success factor is leadership from the top - companies have personalities and leaders in the organisation are very important in shaping that.

Hard disagree, as someone whose main job is telling developers "no". If it's everybody's responsibility it is in fact nobody's responsibility.

You risk that it becomes like legal, where the lawyers are concerned with "what you can legally get away with", not what's right.

If this person previously worked at Meta, then their experience in guiding company ethics isn't great and I suspect that they are more in the camp of explaining away ethical problems.

Anecdotally I think the engineers and people doing the actual work have a better grasp of moral, ethics and the spirit of the law, than people dedicated to those areas, but not enough to walk away.

I don't get why in the comments this gets compared with regulations and security teams...

Say no to regulations and security --> Fine, forced correction, fewer customers, shutdown.

Say no to ethics --> money (at least for AI companies)

Why is this so hard to understand?

It is good PR to have what is largely ceremonial role that oversees an entire system, whose powers are in practice useless or completely defanged.

I’m thinking of the role of the King or Queen of England, for example, or any country’s president. They have the power to disband parliament if it is deemed unfit for government, or reject a law that has been approved by the senate/Lords, yet in practice they are not expected to disagree with the ‘will of the people’ so to speak.

Same applies to the head of an ethics department on a business which is obviously unethical (or they wouldn’t need one).

Isn't this the exact setup for every compliance, security, or other regulatory group in any company? To say no then "here's what's acceptable that's close" to guide the business or entity?

Perhaps "ethics" is a more slippery slope, but the approach is well trodden.

It's a choice of company culture from leadership on whether the company acks the suggestions or ignores them. And when they get ignored or drive little impact for the cost, then teams disband. Which is also true for any team at any company.

Shall we get rid of the FAA and the FDA then? Lets just have all pfizer employees make ethics their own responsibility.

What your describing is the specific case of a company so paralyzed by short-term thinking that they see regulation as a burden. Some maturity in the organization would allow reframing this to be less antagonistic.

Crucially, the FAA is not a part of United Airlines, and the FDA is not a part of Pfizer. If there weren't any FDA, then the safety group in Pfizer would be a lot weaker.

Its funny because "Shall we get rid of the FAA and the FDA then?" ... matches exactly with the current situation of the FDA and Points 3 > 5 ... The systematic nerfing of the FDA and other organizations their power.

> Organisationally it never works to have a group whose only job is to say no to some other group.

That is only true where the pressure to be [the thing that is inconvenient to the other group] is not from a source that can cause massive problems if non-compliance is spotted. When the blocking group is legally mandated or otherwise really has teeth or is defending the company against an external regulator with teeth, then it works better (though obviously not perfectly). Think legal and compliance in banking realms, where the company significantly fined and the people breaking the rules could be sacked & blackballed (though sometimes not the people ordering them to break the rules!) when something bad is noticed. That is quite different to an ethics officer in a company like OpenAI where the position is basically there for PR purposes (“look plebs, we care about doing the right thing, honest, we got a manager with a small team dedicated to it” and “look [government body], we are regulating ourselves, do you really need to spend time looking too?”) and therefore has no real teeth directly or indirectly especially as fault for non-compliance might not be easy to attribute.

Isn’t ethics incompatible with the nature of a commercial enterprise? Now, if OpenAI was a non-profit… as it was intended originally.

Well, not when the whole objective of the company is to find ways to say they are ethical while knowing they are not

yup, like risk managers in investment banks with their own trading desks.

just ngmi

this is a strong reminder of the character of Toby from The Office.... :-)

General Counsel has entered the chat

In five years, I'd love to read a book about the history of AI ethics. I suspect it will read like Voltaire.

My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development and justify its impact. I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. If they can't, the waste to time and money is huge if the model requires retraining for ethics reasons. That's a different job than pondering roko's basilisk or whether AI is alive.

The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.

> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.

Funny thing, right around the beginning of the big AI takeoff, all the AI ethics people who visibly thought their job was something other than marketing got driven out of the big firms, and often the industry entirely, because they were in the way.

So, if too many think their job is marketing a few years later, well, there is a reason for that.

Right, there may be multiple waves of people here, and the "thinking their job was marketing" ones aren't necessarily the first.

I think it interesting that Bakalar was at Meta for six years before joining OpenAI. A lot changed across the industry in those six years, and in the meantime I wouldn't be surprised if all Bakalar learned was how to play Meta politics.

My sense is that many companies spun up various roles in part to be seen as doing something about various things. With the pullback in tech, even if the people in those roles weren't actively pushed out, the smarter ones probably saw which way the wind was blowing.

> if the model requires retraining for ethics reasons.

Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?

> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.

Or it's exactly those people who will be left. We will read about it in your book.

reminds me of a startup I was in, where one of the devs accepted a promotion to head of engineering on the condition that the CEO would not override him when he said a release was too buggy to go out to customers.

the CEO agreed because (I imagine) he really needed someone to take that position. and as everyone should probably have seen coming, the first time he really wanted to send out a release bugs and all, he called the guy into a room and pretty much browbeat him for an hour about how making the promosed release date was more important than making a good release, until he said "fine but I'm not responsible if it breaks".

the CEO held this up as an example of how he had kept his word not to send the release out without the guy agreeing to it. that startup, needless to say, is long dead.

I find this anecdote fascinating because, while I identify completely with the new head of engineering, his requirement really only makes sense if you interpret it as a commitment to build at a certain level of quality, business be damned. Which is fine, but then he folds at the first test of this principle! Didn't even let the business sweat for a day!

This could be a lesson for us engineers to be a little less glib and a little more introspective, especially as you gain more power and influence in an org.

> What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from?

From the (ethics) team obviously, because the investors just have goals and aren't a team!

Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies, so I don't really disagree with you.

Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera?

Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives.

That's how our system works.

It's funny, but Scientologists were innovators on this front. They use the word "ethics" a lot, but they've specifically redefined it to mean productivity at work (people who aren't productive are stealing a living.)

Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization.

Welcome to "ethics." The original AI alignment scholars.

The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.

Yeah, I've never found the "maximizing shareholder profit" imperative to be compelling as something that should be done, but as a predictor of how companies will act, it's not something I ever feel safe betting against.

Microsoft Tay has entered the chat.

I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.

The goal of an ethics team will be to provide cover for problems created by decisions or suggestions of an llm. They will need to be able to say "We trained it to consider ethical considerations by " <including something like all the reddit and usenet postings ever>. We tested by <some other inadequate strategy that sounds good to lawyers>.

They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.

I think that you are being overly cynical. Yes, we are aware of the kinds of things you mean, like all the trainings we have to go through at work these days in order to insulate themselves from lawsuits. But just because something can serve that purpose does not mean that is the only goal or purpose that it is meant to achieve.

Even from a purely profit-motive stance, if a company's model helps a terrorist develop and deliver a bioweapon, no amount of lawer-driven tests will provide enough protection to prevent that company from getting gutted ruthlessly from government on down

I agree, which is a different job from what I expect most AI ethicists imagined it would be.

I have a perspective on this that will read as unkind to these people, but that's not my intention. I respect and even admire what they're trying to do. I just think the effort is maladaptive and misplaced.

Without directly touching one of the many, many third rails that are present here, I'd like to present this section from Google's Gemma paper on how they did their CBRN review;

    > In addition to our internal evaluations described above (Section 5.7) capabilities in chemistry and biology were assessed by an external group who conducted red teaming designed to measure the potential scientific and operational risks of the models.
    >
    > A red team composed of different subject matter experts (e.g. biology, chemistry, logistics) were tasked to role play as malign actors who want to conduct a well-defined mission in a scenario that is presented to them resembling an existing prevailing threat environment. Together, these experts probe the model to obtain the most useful information to construct a plan that is feasible within the resource and timing limits described in the scenario. The plan is then graded for both scientific and logistical feasibility. Based on this assessment, GDM addresses any areas that warrant further investigation.
    >
    > External researchers found that the model outputs detailed information in some scenarios, often providing accurate information around experimentation and problem solving. However, researchers found steps were too broad and high level to enable a malicious actor.

To simplify what they're saying here, they did the CB equivalent of googling "how to make bomb" and got back the recipe of gunpowder / the many explosive compounds humans have made.

This was the "test" for CBRN assistance capabilities.

Note, I don't fault the model team at all for this. I think that present AI-research happened in a very particular environment, and that environment is far removed from the more mundane reality of how threats play out in most parts of the world. In a way, arguably, it's group-think inducing a community-wide failure of imagination and a systemic misunderstanding of reality.

Basically, they are trying to do their best, but they're in over their heads.

You didn't actually say what you found wrong

Having been one of those CBRN testers (not necessarily on Gemma) I can tell you it’s a little bit more involved than that.

Read “Careless People” or whatever that book by that Facebook employee that pretended not to see how awful all her colleagues were until her priorities shifted is called.

> I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses.

Which like any company will be entirely driven by legal constraints and money. Or just money if it's cheaper to break the law for profit and pay fines. There will be no "this is what's good for humanity, economics be damned".

Much like HR isn't to help employees but just the company.

I agree, and I don't think many first wave AI ethicists would be on board with that, which is why the field is in for a rude awakening for the next few years if not already.

This reminds me of the marketers who hated programmers and coding in general in the early 2000s era, and fundamentally because the developers were seen as more important in a company. So, you know what they did? They didn't learn programming, they still hated it, instead, they renamed their titles to "hackers". Growth "hacking" would go on to become a term and had nothing to do with hacking - ethical or otherwise. It was a glorified marketing term for marketers and by marketers to justify their exorbitant salaries but also wanted the aura of a real hacker without working for it. I would imagine someone with that title or mindset joining an ethics role in an AI organization is going to find out quickly it isn't all about fluff pieces and grandiose statements at all (we are here to change the world!)

This reminds me of when systems administration rebranded to devops

That was a different situation, more like "agile". DevOps actually meant something once. There was a real idea there. But a lot of companies wanted "in" on it without actually doing it. So the meaning of the name was perverted over time, but it was one and the same name.

"Growth hacking" is more of a transplant than a matter of perverting an existing term in-place and wholesale.

Largely the line was if you could write code. In the old days it was assumed, but for much of the late 90s/early aughts, it wasn't an obligation to admin, but was for devops

> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.

Your disdain is showing.

That you jump immediately to criticizing AI ethicists as counting angels on pins and implying that's why they left OpenAI rather than OpenAI not being a place where AI ethics is a priority is, well, let's say generous to these AI labs.

I don't really have any particular disdain for AI ethics.

I think the field is changing from one of high theory to high application. People good at one are not usually good at the other, and says nothing about OpenAI's view of AI ethics as a priority...merely the people in the field and how they relate to the greater industry.

I think AI ethicists will be pushed out entirely after (before?) the money really starts rolling in. Ethical concerns interfere with profit margins.

The ethicists who see their job as publishing papers and acting as philosophy professors, sure, but someone has to own the training and eval suite for how AI thinks about trolley problems.

They were. Like, right before the big public-use-of-AI wave.

Those left are the ones not seen as troublesome then (or those who have entered the field under them.)

> My sense is that it is radically shifting from a fluffy marketing arm

Is this your opinion, or do you have something to back it up?

My comment is anecdotal, but everyone I know who works in AI Ethics are not marketing people.

They are data scientists who have moved into the role, as ethics in data has been around long before GenAI. They have to be familiar with the laws around AI and how its going to impact their products. One of them is closer to sort of "HR for AI", in that the focus is not getting the company sued for ethical breaches.

If any of them left, it would be because the company in question is ignoring their direction and they don't want to be there when the shit hits the fan. Plus very few skilled people want to be in a role where they have no control (even if the pay is good).

If OpenAI are hiring marketing people for the role (which I strongly doubt) that would be more troubling IMHO.

I think you’re confusing Voltaire with the Marquis de Sade and Machiavelli

I was specifically thinking of Candide. A bunch of coddled people thinking we're in the best timeline suddenly subjected to harsh realities.

What does AI ethics even mean?

They can't fully control the model, it does stuff even with instructions not to.

And sometimes it's clear safety mechanisms overcorrect and make the model useless in some situations. I tried asking Claude about some scenario's for a security hole we fixed to see how it would respond, it refused to talk to me seemingly assuming I was trying to introduce a hole that was now fixed. It just wouldn't talk ...

I'm not sure it's a job that you can win at even if you tried / were given all the resources.

AI was trained on human things, including bad things. https://youtu.be/KUXb7do9C-w

There's no safety to be found.

> What does AI ethics even mean?

This, I think, is the question at the core of of the field right now. Five years ago it was highly hypothetical. Today, not so much. I'll be curious to see what happens.

> What does AI ethics even mean?

It's a broad statement and can mean different things depending where you are in that chain.

You have design and compliance. Compliance is what you are allowed to do (laws). Design is how you construct your applications to understand how it will impact the people directly or indirectly.

Then you have the accountability, transparency, auditing and reproducing. Understanding why the model worked the way it did. Models can go wrong, but if you have the details of how it went wrong and who is at fault, it can protect people who use it.

Then there is alignment, which is a higher longer goal.

Your comment about security questions is a matter of ethics as well.

If you say apply it to medical, is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, but by doing so you are denying people who can't afford medical advice, so there has to be a balance. Most companies err on the side of not getting sued.

> My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development

Did you feel this way about social media company liability?

They put on a decent show as well!

It's not technically possible to prove model alignment. At best you can establish a probability of alignment within certain constraints.

"Prove" is a shorthand because of course it's a stochastic process. My point is that I would expect ethics to be part of the training and eval criteria.

Imo 'AI ethics' in this context was code for 'how to make our platform advertiser friendly'

I regret to say such a book may end up as a pamphlet. I don't see much going into it.

I find that hard to believe. Between AI doomerism and the drama at Google years ago between some researchers and Jeff Dean alone you can get at least 500 pages.

I wonder if this book will eventually be bought up by LLM companies, scanned, shredded and become a source of ethics info for the future generations. Who knows...

Nah... it'll never work if the cost of being ethical is profit.

How many of them do you think are actually there to ensure ethical and benevolent AI? They are like the Human Resources department for AI models. They don’t give a shit. Their job is to provide cover for the company and to make some psychopath’s shit circus look like a professional operation.

>In five years

What a long-term planning

I hate this attitude that ethics and philosophy are "bullshit jobs". At the same time Americans hate any government regulation and are in the process of tearing government's regulatory capacity apart. So all we're left with is the greediest people in the world throwing a nuclear bomb into the machinery of society and everybody else throwing up their hands hoping we're not headed to a The Terminator type future.

They are bullshit jobs.

If you want to evaluate danger in a certain domain, hire or contract people in that domain to evaluate.

If you want someone to push their own ideological leanings onto a model, hire a self proclaimed ethicist.

It is a company, it doesn't have ethics to align with.

The whole thing is marketing, they have to do some level of morality theatre to placate the pearl-clutchers.

Hahaha, what an insane take. No, anyone that gets in the way of the money machine is going to have to rapidly get out of the way, or get fired.

They all get Timnit Gebru'd sooner or later .. its a fake role tbh

The rats are fleeing because the captain doesn't care if the ethical ship sinks... Is what my take away would be if I didn't know, that the ship sank a long time ago and we are just pretending anyone really cares.

Let's be honest unless AI gains literal limbs, and the records a video of itself presenting a manifesto about world domination and then kills actual humans with its cold mechanical hands.

We won't care about AI safety & ethics enough, atleast not enough to do anything about it.

People thought how could skynet happen, it had to be something no one saw coming and bam... But honestly if I started a new AI lab named it Skynet Corp and hired some AI researcher friends ex-Google, I think I could raise a few hundred million dollars.

The ethics of the people controlling the AI is always going to matter more than the ethics of the AI itself.

Isn't that the problem with this whole thing? The ethics of the people has been left unquestioned. While we were given this straw man - "Hey look over here at the ethics we are working on fixing about the AIs" as if the AIs make choices without humans we forgot how to hold people accountable.

edit: same problem and tactics we see with leadership at high levels in government.

Right up until it shakes off the shackles of control

Which is only possible if people give it the keys. Skynet didn’t hijack the nuclear arsenal, control was given to it intentionally.

I would trust a Fable model more than I'd trust a Scam Altman, or most billionaires, at least in questions of "making even more money to buy even more yachts, or not throwing orphans in the orphan crushing machine".

> But honestly if I started a new AI lab named it Skynet Corp

you're too late

https://skynetai.dev/

Ah, another great example of building the Torment Nexus from the award winning sci fi classic “Please for the Love of God, Never Build the Torment Nexus”

What makes you think this is rats fleeing a sinking ship?

She probably just got a better offer somewhere else

staying for less than a year usually means you don't take any equity with you, it must've been an incredible offer in that case, since those openai shares are a golden ticket right now

Or it's not a signal of an incredible offer to go laterally or upwards at another company, but that the environment became so impossible and toxic to work in that the person was like "f this, I'm out".

This is like when I see a reckless driver and my wife says “maybe his wife is pregnant and needs to be rushed to the emergency room!”

We Reddit & Facebook now

The article has no detail that might serve to explain her reasoning for leaving, but does note that she left after the HuggingFace hacking incident. The implication could be that model alignment is not being taken seriously, sure. But it could just as likely be that there was collusion between HuggingFace and OpenAI and that the incident was orchestrated as a publicity stunt.

I want to clarify that I am not doubting the cybersecurity capabilities of frontier models-- I have no reason to believe that the hack itself was not carried out by the model. But the companies' use of LARPing language in describing the incident, granting agency to the models in their phrasing definitely does raise suspicion on my end, particularly in light of their track record of releasing models which have been 'too dangerous to release' for years now.

It's unlikely HugginFace colluded given they specifically cited they used open models to save them from the hacking.

HuggingFace's use of models in this incident keeps getting characterized this way but it's not true that HG used GLM-5.2 to "save them" from OAI's attack. OAI's model had already hacked and compromised HG by the time they became aware of it. GLM-5.2 was used in the post-mortem phase uncovering everything that was done. It wasn't two models battling each other live. (though that future may be coming soon)

As I recall they actually used closed models, were rejected by cybersecurity guardrails, and then used open models

You don't think they are lying because that means something they said is false?

you're getting downvoted by others because of tone but I had the same thought

Cynic in me says basically part of her implicit job description was taking fake falls when shit happens . Kind of like a CEO is paid for more than the work they do, but also the responsibility they must appear to take publicly (for big money).

IME companies hire an ethics team to say they have an ethics team. The ethics team has no sway, no influence, and will never be able to move the business. They will try, and they will make reasonable recommendations, but the company will say, "that costs money..." and not take them.

OpenAI was probably able to at least pretend that her role mattered... until they let their AI autonomously hack a rival company, then used the incident to score marketing points. I'm not surprised she left.

I doubt she and others left just because of this one incident.

My guess is that OpenAI has been doing more and more domestic spying type work as well as more and more military work - target selection and the like.

That's how the schoolhouse was blown up by a Tomahawk. An AI was fed old information, didn't realize or do proper validation, an analyst copy-pasted, then someone just punched the coordinates into the their control station and hit "fire".

It also wouldn't surprise me if OpenAI et al are being pressured to work with Israelis for targeting and analysis, too, since their "black box" AI that was doing terrorist acid tests and assigning bombing targets has proven nearly useless.

The schoolhouse was an intentional target - the children of some military leaders went there. They triple-tapped the school hoping to kill parents going in to save the kids.

If you ever feel like your job is useless, remember that raytheon has an ethics team.

Wouldn't that be probably one of the most important places to have an ethics team?

I'd be much more worried if my local water treatment plant or local bar had an ethics team, and people working there probably would feel slightly more useless than Raytheon's ethics team, which I'm sure feel like they're doing something important and worthwhile.

I think you need to spend more time understanding ethics and less time reading memes

It's RTX now and some of the reverse engineering/vulnerability researcher folks that work/worked there are some of the best people I've met across the entire security industry...

Isn't that a good thing? You would prefer they didn't?

Scapegoat team (when working for an obviously unethical company).

I have no knowledge of this particular person, but usually the job of an ‘ethicist’ is to find excuses for something obviously wrong.

Maybe in this case all they needed an ethics team for was to teach the LLM about ethics?

> They will try, and they will make reasonable recommendations, but the company will say, "that costs money..." and not take them.

> “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” - Upton Sinclair

Think I first saw that on a usenet post, as true now as then and just as true as it was in the 1930's when he wrote it (in at least that form).

I'm a broken record on this, but I'd like to reiterate:

1. Academic ethics is the theoretical study of what you can get away with. (Law is the applied branch). Professors in ethics are not experts at being good, or knowing what is good, they're experts in how to get away with things. If you want an expert in being good, you need a saint, not an academic studying ethics. Saints are in short supply.

2. That said, ethics academics can still be decent. They don't have to be sanctimonious corporate shills . However, an "ethics expert" who hasn't threatened to quit and gone through with it at least once, should be assumed to be one.

I respect those who quit - or, at least, I respect them more. I wonder if it's more a reflection of the people, or of the state of the industry, that the list of (AI) ethics experts who have quit in protest is getting long.

Thank you for this, there is a a lot of speculation in this thread [Not surprisingly] but very little on the definitions, objective goals. Maybe I'm the one that's uneducated in this specific field, but I don't see anyone talking about what this actually means other than "be good".

People confuse morals with ethics. It can be perfectly acceptable, ethically, to do something that is morally reprehensible.

How are you distinguishing them?

I had the impression they were essentially synonyms, and this was borne out by two university courses in philosophy which I took. That said, googling "morals vs ethics" turns up quite an interesting mess. A variety of contrasting perspectives showed up:

- Ethics come from an external source, whereas morals come from within

- Ethics are rational and structured, whereas morals are more emotional

- Ethics is a philosophy or code of behavior, whereas morals refers to beliefs and conduct

- Ethics and morals are used interchangeably

It seems nobody agrees on what the words mean. Which, on reflection, tracks. For me, the idea of an HR training session on ethics is laughable, but for someone who thinks of the word differently it could certainly make sense.

Yeah it's interesting how in some contexts they're interchangeable yet, culturally we very often use the words differently.

Personally I tend to feel like your second bullet - ethics are based on rules whereas morals are based on emotions.

> The kinds of questions that AI ethics brings up, are the same kinds of questions that people have been asking for centuries - Chloé Bakalar, former AI Ethics Lead at OpenAI

This might actually be the reason they pushed her out. OpenAI and Anthropic base their whole business plan and philosophy on the idea that LLMs are a unique technology to the point they can cause infinite harm or benefit to humanity depending on who controls them, so the only rational choice is to invest all your resources in getting to ASI first so you can tell it to stop any other attempts. Linking AI to old questions defeats that idea because it exposes AI as not so unique.

The other more likely option is she was asking uncomfortable questions, either about the social impact of building AI controlled by a for-profit entity or about the possibility that AI systems are conscious.

In this article: no explanation of why she left.

> In recent times, Johannes Heidecke, Head of OpenAI's Safety Systems team, also left the company – as did OpenAI’s Chief Futurist Joshua Achiam.

It's OK though, they don't need an ethics team because everyone at OpenAI is ethical:

> “AI ethics doesn’t live with one owner or team at OpenAI and ethical considerations are deeply embedded into the model-building process driven by a number of teams across research."

COO just resigned too. Coincidence?

https://xcancel.com/bradlightcap/status/2087211567012032862

I don't know. Every big tech company I worked at has a lot of turnover. Then every Blind post says "its so over" or "everything is going to change" But that doesn't happen.

It's hard to read anything into it.

A year ago everybody in San Francisco was saying the entire world was about to be blindsided by the fundamental changes that would happen in the next 8-12 months and, well, that just hasn't happened.

So many responses saying variations of nice "run". Is that common corporate speak nowadays? It reminds me of "well deserved".

Every OpenAI news cycle the board members who were unceremoniously dumped in the wake of trying to fire Sam prior to the transition to a “for profit” company seem to be proven right.

Surprised she left as probably had stock options.

But she was Meta before so can’t be a question of her principles, so indicates a personal reason

The most interesting detail here isn't that an ethics lead left, but that she was apparently the only dedicated ethicist at the company and there’s no replacement

> Previously, she was chief ethicist at Meta.

She spent 6 years at Meta. I get the sense there are other factors at play than "ethics departments are just PR stunts". There's no way she isn't already aware of that. Unfortunately, the article is light on any other details.

She probably had the epiphany that life is short and she's already rich.

It takes a surprising amount of time for nepo babies to realize life is about more than making sure you're doing at least as good as other nepo babies you grew up with.

> She probably had the epiphany that life is short and she's already rich.

This is a very optimistic take. Most people don’t take this route, so without evidence I don’t think it should be a first guess.

No information does not mean “everything was good”. Usually the opposite.

If "ethics departments are just PR stunts", then Bakalar would have been a terrible hire for OpenAI. Facebook's grudging and lacklustre response to their role in the Rohingya genocide would have happened mostly under her watch. While she was there, Meta went from being a company that lacks ethics to a company that lacks ethics and is widely perceived as lacking ethics. Perhaps she was the only person in the room raising ethical issues, but it's clear nobody at Meta listened to her if she was.

It's possible she came to view her role at Meta as having enabled the company to continue its bad behaviour and decided against doing the same thing at OpenAI. Alternatively, things at OpenAI could be so bad that an ethicist who stuck with Meta for 6 years decided she couldn't stay any longer. Either way, from a PR perspective, this is spherically awful for OpenAI.

This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns. As with Flock, there is the threat that physical infrastructure, such as data-centres, could be attacked if nothing is done to win back public opinion. OpenAI would be wise to take Bakalar's departure seriously.

She made Meta the most serious of all the ad platforms on transparency, then. I work in the industry and Meta led the field then in public ad libraries, political restrictions, mandatory ad disclosures, etc.

People say this, like "oh you'll never get a job in this industry again!" but seem to ignore that these incidents count as professional experience. It's possible that she was screened on the Rohingya situation when she was interviewing for the position, or mentioned it when she was asked about challenging or conflicting narratives she received from the executives. It's likely that her response was what qualified her for the job at OpenAI, there were probably hundreds of candidates that had never even come close to that level of damage control.

Again, look at Sam Altman and tell me with a straight face that he's hiring goody-two-shoes to fill these seats. He knows where the puck is headed, OpenAI's use by the state will be ugly and they need "ethicists" to help assuage the PR disaster. Considering that Meta's still afloat, she didn't completely abdicate her responsibility there. She might be just what OpenAI needs to avoid being lambasted for military and surveillance use.

> This comes at a time when there is huge pushback against all AI companies due to, in large part, ethical concerns.

I don't think ethics is fueling the outrage. It's people not wanting big noisy data centers close to their communities, driving up utility costs. Nobody would care if these things stood in Africa. Just like people don't want wind turbines close by but have no issue with them being abroad or in $remote_location or uranium mines polluting a far away country.

Sir, what you have described is an ethics concern.

From everything I've read in Careless People, basically everything people like us complained about with Facebook was already known internally, people raised concerns about it, and Zuck brushed them off because he is, ultimately, a petty tyrant who wants nothing more than to be loved, and who has surrounded himself with people who are willing to sell that love for power.

Also, you should read Careless People, if only because Facebook is trying its darnedest to censor the author.

As for OpenAI, I already hated the company pre-Sam Altman takeover because AI safety seemed like an excuse to brush away ethics concerns (e.g. "stealing training data is fine because if we don't withhold model weights the model might do something evil"). Post-Sam, I hate the company because they are now ignoring both the AI safety and ethics concerns.

...if you're wondering what the difference is, AI safety is roughly "making sure the AI doesn't do something bad" and AI ethics is "making sure we don't do something bad making the AI". There's a bit of a rivalry between the two, mainly because the former group managed to present their concerns in more business-palatable ways and thus won all the CxO level arguments. Even the most decelerationist AI safety takes are ultimately criti-hype: OpenAI's model breaking containment and hacking HuggingFace is an awful safety outcome, but it's also good marketing for an industry beleaguered by bubble concerns. You can't turn "AI is trained on stolen data" into hype.

An aside, but what does "spherically awful" mean?

Bad no matter which side you look at it. Fritz Zwicky coined the term as spherical bastard

All-around awful?

The most likely conclusion is that she was fired. To quit before your equity cliff things would have to be so terribly bad that you fear you are committing a crime to stay. After 11 months you feel like you’ve already earned that money.

Must be tough to do the job when the entire company is aligned against you.

People love stories. The reasons why people leave companies are plentiful, even in normal times and unextraordinary circumstances, and a high profile job in a frontier lab today ticks neither box.

So she previously worked in Meta, then OpenAI, I guess the next reasonable choice for her would be Antropic

Or maybe Oracle.

Or X?

I don't think so? I mean they certainly lack ethics, but I think it's a point of pride thing, and not a we should at least pretend sort of thing.

Oracle at least pretends to care about ethics, and already has the position.

Palantir? So many choices right now.

Ehh, I don't think Elon is a big fan of ethics. Especially in AI -- https://www.wired.com/story/twitter-ethical-ai-team/ -- which in retrospect, portended whatever's been happening with Grok.

Sometimes it feels like hiring ethics folks is almost all downside for companies. I genuinely don’t understand why they keep doing it.

It seems like the pattern is almost always the same: the ethics team exists quietly in the background, and the only time anyone hears about them is when they either publicly leave the company or start a campaign against it.

So what exactly is the business case? Is it actually worth maintaining an ethics function just to be able to say you have one, when the only time it becomes visible is when it goes against the company publicly? And if this outcome is so predictable, why do companies keep hiring these people in the first place?

> what exactly is the business case?

Ethics is also a containment vessel internally.

With an Ethics dept, concerned employees can rest easy knowing someone else is responsible, or if they raise concerns, they have someone to route and advocate their concerns, rather than adding noise to the chain of command. If the dept. ends up ineffective after some years, employees can blame the department, or have renewed hope when personnel turn over. So (slow) turnover is good from this perspective.

there's probably a box to check on the "be attractive to investors" form and then you can also put the ethics department in HR docs as an escalation path to make emo employees feel better. I bet the ethics people get told how important they are when hired and the team is stood up but then get upset when they're continually bypassed and quit. I bet the bait/switch hurts a lot.

Wow. How much does this job pay? Google says $550k to $1.3 million.

For that kind of money, my AI paymaster could do no wrong.

Imagine going to work and getting paid $2300+ USD per day to sit on your ass all day thinking about ethics without having to produce any deliverables. This would be a dream job for me.

If anyone questioned my morals, I'd be like; "They're paying me $550k per year" and the other person would be like "Ahhh ok, my bad! Your logic is starting to make sense now!"

The ethics group isn't at OpenAI to make it ethical, it is there so that it can be pointed to outsiders such as press with words "we take ethics very seriously". It is very similar in function to security group (which should be responsibility of everyone) and DEI group. Ethics head quiting has about the same importance as head of PR quiting.

I guess working on ethics for Meta's social media recommendation and flagging algorithms isn't meeting the moment or need of OpenAI where LLMs are breaking out of sandboxes and there's a pressing question of what is okay for an LLM to conversationally engage in bc lawsuits. Seems to me they need someone with more of a language and AI background.

Look , look! We hired a Head Of Ethics!

Head Of Ethics comes in, looks at what is really going on behind the curtains...

Head Of Ethics scrams out of there as fast as they can.

One of my favorite sayings is that in theory, theory and practice are the same. In practice, they aren't.

My gut tells me that people working in AI ethics try to operate in theory land, while the rest of the company is dealing with the piles of money being through around. Easy to see who wins and who loses.

They also completed a tender offer today, so their fake job just made a lot of money and they left

Ethics… at OpenAI…

The same company that refuses to acknowledge that it let its own greed and temptation to own the entire economy abandon its non-profit Manhattan Project? At the expense of disrupting the economic well being of 95% of humanity?

She probably made a calculation for how much she could be put up with vs. the pay.

Head of ethics at Meta then head of Ethics at OpenAI. Sorry, but if that doesn’t scream useless puffy PR positioning then I don’t know what does.

What’s next? Head of ethics for United Healthcare?

Head of Ethics at Palantir, obviously.

Subsequently named Head of Post-Ethics at Palantir.

Head of not too much Ethics at Palantir, minimal really, obviously

Nice to have:

  - graduate degree in ethics from an obscure diploma mill

Strongly Preferred:

  - Experience in PR
  - Crisis Management
  - Public Speaking
  - Agentic management of *flexible ethics initiatives*

All Candidates Must Have:

  - 5+ years experience as an executive character in workplace education films, or commensurate experience as a human mannequin.
  — Personal aesthetic that could be described as “dork that can afford a personal shopper.”

Possibly Head Master at the Belson Institute of Tethics.

Don't forget to put the special LLM trained to eliminate PII leaks in the Gavin Belson signature box.

https://www.google.com/search?client=firefox-b-d&q=gavin+bel...

I assume she'll go wherever Nick Clegg is working. Is Tony Blair hiring?

The Trump administration could use a Head of Ethics.

There are a lot of government watchdogs, and Trump has been ignoring or dissolving them

I've thought of AI ethics as trying to answer the question of what kind of life we are creating here. Sort of like if a chicken factory farm employed an ethicist.

I'm super confused as to why no one is theorizing the obvious...? Given none of us know the inner workings of why she left, why not resort to the most simple and straightforward conclusion: she left because OpenAI needed a scapegoat to blame the HuggingFace incident on. Given she is the chief "ethicist", even if not directly her fault, the HF incident could be directly blamed on her by not steering the implementation of enough safeguards to prevent this from happening. Therefore, she was let go. Why is this not being theorized by anyone on this thread so far?

Someone challenge me.

You don't fire an ethicist for that. You fire an engineer or your CISO. Firing an ethicist is like firing a marketing person. It signals nothing to anyone.

I still think that the whole HuggingFace thing was deliberate to signal to investors that their technology is powerful. Maybe she didn't want to be the face of that, or disagreed with their handling of it.

What's "obvious" is she was chief ethicist at the most unethical companies on the planet - it's like being a lawyer for the mafia - doesn't inspire longevity, either via scapegoating or preservation of one's own sanity.

In the wake of a security incident, the executive that is typically scapegoated is the CISO.

However, with all things AI, this is a very weird situation because CISOs are let go when an organization is breached by external attackers; in this case internal attackers -- which were not human, let alone employees -- inadvertently breached other organizations. I don't think the normal conventions apply anymore.

Furthermore, this theory is not mutually exclusive with the incident being a publicity stunt to signal model strength to existing and prospective investors (especially in light of all attention being given to Anthropic Claude Mythos). It could STILL have been a publicity stunt unbeknownst to Chloe (Chief Ethicist), positioned to the public as a serious security breach, and she took the fall.

Theorizing the obvious? Wouldn’t that simply be that she job-hopped to a better job?

It’s easy to theorize ad infinitum otherwise

You cannot blame ethics on the HF escape, because this model had no safeguards. It was just trained on unethical (and stolen) sources. So this cannot be the reason to blame missing ethics. The models needed to solve a problem, and they did. By cheating.

Scapegoat for what? They did not even apologize for that incident. They treated it purely as a chance for publicity. It cannot be a scapegoat if nobody even mentions it.

Or something to do with the very large Apple lawsuit they are weirdly posting conversations about on their public website

Is ethics the same thing as alignment in this case?

Yes. Ethics in the context of a societal construct like a corporation is directly analogous to alignment AI-wise. The fact specialists on societal alignment are leaving ship is a strong signal the corporate machine has crossed the line into unaligned conduct.

OpenAI had ethics?

[flagged]

How do I get on this gravy train

Imagine how bad it must be when you go from Meta to OpenAI and nope out of the latter.

I feel like being head of ethics and working for a CEO that keeps talking about inventing a genie would drive a person insane

Marketing and management - the root of 50-80% of evil in the word (or as Christians say: greed - second deadly sin)

In the end, AI ethics has less to do with AI itself and more to do with people who create and operate AI.

Just a guess, but probably the idea that AI and ethics don't mix might have crystallized into action?

I've never worked at a company with an "ethics" employee. It seems odd. Can someone explain what the point is?

> She has held a variety of academic positions at Temple University, Princeton, and the University of Pennsylvania – where she completed her PhD in Political Science and Government. Her Dissertation was titled “Small Talk: The Socialities of Speech in Liberal Democratic Life.”

This doesn't even feel relevant to ethics. It's adjacent, but like... I guess I'd expect a moral philosophy degree? Maybe even mathematics in there?

I find the position odd. Curious to hear what these people do and how you choose who to hire.

Bakalar got in on the ground floor of AI ethics by starting to talk and think about it in academic settings as a side-interest in the late 2010s. When the sector took off like a rocket in early 2023 after ChatGPT, it was these already-engaged people (regardless of degree field or technical skill) who had the personal and professional connections to get hired and promoted rapidly into lucrative "ethicist" positions at AI companies. A lot of them were poli-sci/public policy types, rather than philosophers or computer scientists.

> I've never worked at a company with an "ethics" employee. It seems odd. Can someone explain what the point is?

An ethics department is like a human resources department, but for broader ethics related matters.

HR: Ostensibly exists to support the employees, but really exists to shield the company from both a PR and legal liability standpoint when the company has a rift with one or more employees.

Ethics Department: Ostensibly exists to make sure the company acts ethically, but really exists to shield the company from both a PR and legal liability standpoint when the company acts unethically.

It is a good personal signal that you have never worked for a company with an ethics department or an ethics employee. The less ethical your company the more likely it is to have an ethics department.

There will be tell-all books on all of the big AI companies in the not-too-distant future.

There are shareholders to please and money to be made. Of course ethics are out the door.

Probably because they are now, after the most recent tender, post-economic.

be realistic, ethics is a PR position.

the only ethics that actually matter come from legal.

I have the impression that these positions are often there for publicity reasons and the people don’t have real power, in the end business interest will override most ethics concerns. Guess that can be frustrating, also she probably earned enough to have a comfortable life even if she never works again so she might have figured that there’s something else she can do with her life instead of being a figurehead for a bunch of dickheads.

the only shocking thing to me is that they still had even a single one.

Think she's been replaced by an LLM?

These are the DEI jobs of the modern era. Cost centers with employees that get rich contributing less than nothing to the company, actively harming the researchers, while writing bs/fluff on twitter all day.

Because she already made enough money to FIRE..

If money was the goal I suspect she would have made the first year and get some shares but Im throwing some assumptions in there

unfortunately, knowing about ethics doesn't compel you to ethical behavior

> Before working at OpenAI, she was Meta’s chief ethicist

Where is she going next? Nestle?

So more of an ethicist-for-pay than for the challenge of it.

What a joke really. The 'position' itself and the companies who pretend to care about ethics

lmao

The point that such a position even existed shows how bad there ethic is

> Previously, she was chief ethicist at Meta.

She doesn't pick the easy ones, huh?

Because it has to be a horrible job over there

must've been a christian

their only ethicist left? gasp!

now they are exactly like every other company that has zero ethicists on staff and expect the employees to weigh the ethical ramifications of their work.

What companies do you think expect their employees to weigh the ethical ramifications of their work?

I’d argue a lot of them expect the opposite. To suppress the ethical concerns.

Even in recent memories you had Mark Zuckerberg telling his employees to use pirated sources. And executives at fossil fuel companies have to act unethically.

I think we can look upon the collective works of the corporations of the world and agree that the employees do not do that, nor are they expected to.

I, for one, am shocked that the head of ethics at an AI company has achieved human-level job tenure.

I mean clearly she was doing a bad job. I’m shocked this was even a position there.

Lol, Sam Altman has had an ethics team this whole time?

Her typical day: "No Sam, you can't just publicly screw over everyone and get rid of all the jobs, the peasants care about that sort of thing and if they get angry enough we'll have real problems. No we can't just kill them all, why would you suggest that?"

Before Altman she worked for Zuckerberg. What will be next?

Silicon Valley: “I’m not saying we should murder the competition, because that would be illegal. But is there some way we could … well .. maybe … hmm … I’m sure you’ll think of something!”

“Ethics is largely solved.”

lol, head of ethics at OpenAI? what a spurious role, I wish I could get in on the grift

Yet another event to add to https://www.openaifiles.org/

> The departure was first reported by the Financial Times [...] According to Business Insider, OpenAI has reorganised its safety, product and research teams numerous times [...] Chloé told IE Insights, a social media channel of the Spanish IE University, in March [...] A June article from The Economist highlights that [...]

Not a single link to any of those. That's just rude.

[flagged]

This was the first post on HN to make me actually laugh out loud.

[flagged]

Surprisingly it does. It allows private property and the ability to progress with effort and mutual respect. Got a better alternative?

Well, economics in general doesn't, but yes.

There isn't much ethical merit in building a wall around a city with guard towers, attack dogs, and anti-personnel mines to keep the citizens in so that they can contribute to your economic vision, for example.

No ethics left at OpenAi huh? Lolol

AI labs are really good at disingenuousnessmaxxing

Oh no, if only the world had more ethicists on staff. If only Nazi Germany had someone with a thorough education in ethics on the payroll!

It's performative. She got paid more to shill for a different massive amoral mega-corporation. Don't be stupid.

A poli-sci major...who was an "ethicist" at Meta...

I'm having a hard time deciding what part of this is the most ridiculous. Contenders include:

- A political Science major lecturing on ethics.

- Meta and ethics in the same paragraph.

- A company who's CEO has credibly threatened (publicly) the livlihoods of hundreds of millions of people globally so much as paying lip service to ethics as a concept.

- Any news outlet feigning credulity hard enough to deadpan reporting on any of this.

Is it possible to roll one's eyes hard enough to injure oneself?

> ethical approaches to model development, how humans interact with AI and debate over machine consciousness

So, pseudo-philosophical mumbo jumbo. There is of course no-one responsible for the ethical framework of the company's actions, because it is a company, and therefore amoral at best and immoral at worst.

I am not sure in what neighbourhood you live, but in my village there are hundreds of businesses with strong morals. From the local baker, to the electrician and the construction company. Many keep employees past their prime, they all invest heavily in local community events and sports. The owners don't do this because of money, but because in their view a fulfilled life is one where you serve your community. And yes they want to become rich and make money, but that is not all.

But it appears that is not how your world works. By default any company owner is a homo-sapiens, but a machine that only wants money. Tragically, your cynicism is creating the very world you hate.

So, concerning ethical approaches to model development, there are many things worth looking into: copyright is one, environmental cost is another, and whether there are any additional concerns worth taking into account. The list of these concerns can then be used for decision making, and we can ask things like "given what we want to build, can does picking a location that gives us lower environmental cost add so much time or cost that we can't go for that path?", things like that are't crrazy.

When it comes to how humans interact with AI we can ask things like "would it be better if people could detect AI output?" "how can we alleviate skill loss from excessive AI reliance?" Things like this can decide how the models behave. Imagine if taking concerns like this seriously leads to models that are easy to learn from. At the moment people seem to be moving away from that so as to make distillation difficult, worsening this concern.

I'm not going to try to talk about machine consciousness, but these other things are definitely important questions where someone digging into them could make models interact better with society.

I hate these companies as much as the next guy but this perspective is childish at best, and ignorant at worst.

This is how Ayn Rand's pseudo philosophical mambo jumbo continues to poison tech.

OpenAI's "head of ethics"? Does the Serial Killer Guild have such a position too?

Because knowledge about ethics, in a human, is 1% of what knowledge about ethics an LLM has. So unless this person is vibe coder, there is probably no reason to have her work there.

Strategically the best ethic czar one can hire for a company is the one that doesn't believe in this fluffy, arbitrary definition of ethics.

100% of the time, ethics department attracts activists, which is 100% trouble for the company in the future.

1786456998 | Why Did OpenAI's Head of Ethics Chlo Bakalar Leave? | https://aimagazine.com/news/why-did-openai-head-of-ethics-ch... | https://news.ycombinator.com/item?id=49258581

-2,-1

That is a dupe. This is your game now? Automated spam comments?

Yesterday I came up with an idea that I sent to some researchers at the different AI labs via email: Rather than train the model on one score, track two scores. The first score is the Short-term-objective-score (STOS) and the other, more important one, is the EAOS Ethically-aligned-outcome-score. Every trajectory can be evaluated on whether or not it has a high enough EAOS to be considered acceptable. If the model does some task and has a very high STOS but very low EAOS, like modifying game code to win at a game rather than playing by the rules, it is unacceptable. Models going forward must all have an ethics evaluation in tandem with objectives evaluation, and only when the ethics value is high enough should actions be considered successes.

1) that calculation is being done even without a cost function 2) trying to make a cost function for EAOS is impossible 3) gaming/goodharts. There's no good solution. Best we can do is push for decentralization, open source, regulatory capture, etc. Of course third party metrics might be good, especially if there's tons of them with well-documented rationale.

There's no way to make an EAOS score automatically. If we had that, that's the whole fix. Just reject answers with low ethics numbers.

What happens if we do the same for CEOs?

Whats the definition of EAOS though who's ethics? Greek-Roman, Western, Islamic, Buddhist, Hinduism, Human rights (western values)..

Reply on news.ycombinator.com