Helen Toner shares her side
wsj.com
wsj.com
The embarassing part is that the board decided to fire the CEO, announced their decision, refused to say why, attempted to put in place a new CEO but had to immediately demote the new CEO (Mira) after she rejected their plan, upset and alienated their core partners, along with almost all of their employees, and then publicly backtracked to undo the firing that led to this all happening.
Once you screw this up to such an incredible degree, how can anyone really trust that you were doing the rest of your job well?
Perhaps there's some level of overconfidence at play from systems thinkers who overintellectualize their ability to conceptualize and extrapolate forward an impossibly complex system.
You may well be on to something. I'd trust a cabal of science fiction writers more than that I would trust these self appointed governors of our collective future. They lack imagination, for starters.
- Heinlein
https://www.spectator.com.au/2019/03/robert-a-heinlein-the-giant-of-sf-was-sexist-racist-and-certainly-no-stylist/
https://www.publicbooks.org/asimovs-empire-asimovs-wall/
https://www.vice.com/en/article/bjxp5m/we-asked-people-what-childhood-moment-shaped-them-the-most
(nebulous, but credible)
https://en.wikipedia.org/wiki/Orson_Scott_Card#Homosexuality_2
I have book by all of them on my shelves but they're not necessarily saints.
More importantly, if you don't like somebody, don't just hint at it- give us a link.
He was completely into turning into a woman even early on much less the book where he had his brain put into a secretary who died in an accident and went into the ramifications. I say he as Lazarus Long and the rest were him.
archive.xxx is prblematic at best if you use Cloudflare 1.1.1.1 DNS, fine with other providers, Quad9, Google, etc.
What authors are approved for reading?
I want to make sure I fit into the perfect cookie cutter world views.
Thank you for that, it had completely passed me by.
Oh and the fosterites were poking fun at Scientology, Mormons, and proto megachurch People.
And his intent wasn't to rip off or lie to anyone
Actually this is more or less the point that Eliezer Yudkowsky makes in this essay about the need for caution in AI development:
https://www.econlib.org/archives/2016/03/so_far_unfriend.htm... (see especially points A/B/C/D)
I doubt overconfidence is a problem specific to effective altruism. In any case, any good machine learning engineer knows that a dataset with only a single data point is essentially worthless -- even if we grant the premise that the board took the wrong action given the information they had available to them at the time.
Of course the recommendations are not that novel; that's CEO succession planning 101. But I guess none of those four have done any large succession planning, and were clearly out of their depth.
[1] https://chat.openai.com/share/e77bc868-fe27-4346-9c13-0908e8...
> how one decision would pan out
I'm not sure what point you are making here. Are you trying to say "see, the AI not-kill-everyone-ists couldn't predict the future even in the short term, therefore we shouldn't put much credence into the the idea that the specific examples of AI doom they have given will happen"?
Or are you trying to imply that the idea of AI doom as a whole is bunk, because we can't predict the future... therefore everything will be fine...?
According to reports, Mira was informed of the decision to fire Sam the night before Sam was fired. There's no indication she was against it and she also didn't inform Sam. So, I don't get this "she rejected their plan".
If the board informed her they were going to fire Sam, it seems logical that they also informed her they were going to give her the interim CEO position. I have no knowledge of this (it just kind of makes sense to me).
Are you saying they told Mira they were going to fire Sam and they didn't give her a reason but she was still okay with it?
Mira immediately started trying Sam back. [1]
If that's not rejecting the board's plans, I don't know what is.
[1] https://economictimes.indiatimes.com/tech/technology/openais...
As someone who pretty harshly described the board's actions, I agree with the first part of what you wrote, but I don't agree with this.
People can be good at critical aspects of their job while lacking the maturity to understand large group dynamics. I thought this quote from the article said a lot:
> The board members weren’t prepared for the fallout from their decision. The members, including Toner, were taken aback by staffers’ apparent willingness to abandon the company without Altman at the helm and the extent to which the management team sided with the ousted CEO, according to people familiar with the matter.
It's pretty hard to me to think, based on the board's initial press release, that there wouldn't be a huge fallout from this decision. The release also made me think that Altman had done something truly egregious/malevolent, and that wasn't the case.
What I saw this as was a group of people who had real concerns about Altman's leadership, and some of those concerns ended up sounding pretty valid, but who didn't have the experience/maturity to understand how to go about the next step. OpenAI's "frankenstein" org structure, with a board beholden to the nonprofit's charter but also responsible for billions in investor capital, also helped contribute to this cluster-f.
What did you hear that sounded valid? Toner had her chance to make her case here, and it was simply that Altman tried to get her fired, so she retaliated. It wasn't about safety or leadership. It was petty politics over board control.
No, but lying to Board members about what other Board members have said is. The employee equivalent would be lying to one member of the C-suite about what another had said.
Yes. He's not a teenage girl, he's the CEO of a nonprofit. At his level this is fraudulent misrepresentation.
The rack up debt and fail on mediocre idea and then sell your only valuable resource, your human employees Plan gave him a lot of money to hire a lot of law school grads who have since made partner.
I don't think it's that simple.
First, after confronting Toner directly and getting a response he didn't like, Altman went behind her back to the other board members trying to get her fired. That's not what a responsible CEO does. A responsible CEO, if he does not agree with the position taken by one Board member, brings that concern openly to the rest of the Board in a formal setting. He doesn't go behind the one Board member's back. If Altman really believed Toner's actions had harmed the company, his responsibility as a CEO was to bring that concern openly to the Board and let them make the call openly. And that's especially true for a company that, according to Altman's own public claims, has stewardship of a technology that could be an existential threat to humans. That makes any behind-the-scenes maneuvering on his part even more of a problem.
Second, according to the article, other Board members had concerns about Altman's honesty, and not just due to whatever transpired during Altman's behind-the-back attempts to get Toner fired. So it's not just that particular incident that's relevant here.
That's not to say that the Board didn't make mistakes. I agree with other posters that they should have seen the consequences coming for firing Altman. It also appears, from the fact that the lawyer's advice about fiduciary duty came after they fired Altman, that they didn't seek legal advice before doing that, which seems like an obvious mistake. (It's possible, though, that they did ask the lawyer beforehand and he didn't raise the possibility of a backlash and consequent damage to the company, which would put at least some of the blame on him.) I just don't think that because the Board messed up, that makes Altman the good guy. I don't think any of these people come out of this looking good. Certainly I don't have confidence in any of them as stewards of AI if AI is really as big a deal as they think it is.
Yes and no.
Meetings are a terrible way to drive toward decisions but a great way to signal positions to others. Or "decision meetings" bad; "consensus meetings" good.
If you're driving one of those, you should know how everyone involved feels before they meet. You should lobby any of them beforehand if necessary. The idea is to try to make sure you have buy-in before everyone shows up - no drag-out arguments, no rabbit trails, just statements of their positions and a thumbs up.
And Altman seems to know that, given his behavior.
The decision Altman was driving for was to fire a Board member he disagreed with. That's fundamentally different from an ordinary Board decision, and should not be lobbied for in advance the way you would organize support in advance for a normal Board decision. When the CEO of the company has a fundamental disagreement with a Board member, and wants to get that Board member fired, lobbying beforehand behind the one Board member's back is fundamentally dishonest. He should put the disagreement before the entire Board openly.
Btw, if I were Toner, as soon as I knew I had a fundamental disagreement with the CEO, I would have told the other Board members and requested a Board meeting to discuss the matter (and told the CEO I was doing those things). The meeting would not have to be an actual decision meeting with the Board formally in session. But I would want any discussion to take place with everyone present and with everything openly out on the table, not in behind-the-back individual sessions. I'm somewhat surprised that Toner didn't do that.
I agree with this as a general principle, but in this specific case, the 'understanding group dynamics' part is the 'critical aspect of the job' part. By the same token, you don't just promote the best engineer to CEO if he's bad at managing, because the managing is the critical part of the job.
That and that alone is enough to question their competence, though. The impression I get is that Toner has been in the academic/nonprofit world for so long that she doesn't understand how the real world works. In those places, the sad truth is nothing you really do is of much consequence to anybody in the grand scheme of things, other than you collecting prestige and a paycheck.
Then she tried making a consequential decision in an organization valued at around $90 billion, and lo and behold, people started caring about her actions in a way she has never experienced before.
How can anyone take any "AI" company seriously other than as a get rich quick scheme. Doubtful a successful investor focused on management would bother with a company like OpenAI.
https://fortune.com/2023/05/06/ai-warren-buffett-charlie-mun...
The probability of OpenAI employing wackos, like the guy at Google who thought automplete was sentient, is very high.
Not one critic I've seen has stated what the best legal structure for OpenAI should actually have been, in a world where OpenAI really is stewarding a technology that could be an existential threat to humans. I really wish they would do so, because then we'd have a shot at improving OpenAI's governance. But so far it's been crickets.
BTW, I think one of the big lessons of this episode is that legal structure matters much less than you'd think. On paper, OpenAI is supposed to follow its charter https://openai.com/charter and the board had the power to fire Sam. In practice, Sam was kinda sorta violating the charter, but he's a charismatic leader who was about to bring in a big payday for the employees (https://www.cnbc.com/2023/11/30/openai-tender-offer-on-track...), and that turned out to matter more than the org's legal structure.
Here's a group pushing for regulation and looking for volunteers: https://pauseai.info/
I would hesitate to physically break stuff -- sounds potentially unethical and could harm momentum for regulation
However I make an exception for existentially dangerous technology
Instead of a structure where the board members get to choose other board members, I would suggest a structure where different outside groups get fixed numbers of board members. So, for example, it could have been (remembering that the initial board was just Sam Altman and Elon Musk):
* One board member chosen by Sam Altman
* One board member chosen by Elon Musk
* One board member chosen by OpenPhil
* One board member chosen by Microsoft
* One board member chosen by some VC investor
The precise composition is something they would argue about during funding, maybe some of these groups would get more or less seats.
The root "governance" problem is that the board churned a lot because so many people had conflicts, or other reasons that I don't know as an outsider. The remaining board was unrepresentative of stakeholders (ie, no Elon Musk, nobody independently trusted by Microsoft) and probably not as experienced as you'd like. The problem with this sort of structure is that it gives everyone a lot of incentives to play politics with the composition of the board itself. Which happened a lot here - Reid Hoffman exiting and not really getting replaced by a similar person, the debate between Sam and Helen about her presence on the board, probably other cases.
I bet they end up with something similar, with different stakeholders represented. Give Bret Taylor some time to cook.
I'm reminded of these great PG tweets:
>When people criticize an action on the grounds of the "optics," they're almost always full of shit. All they're really saying is "What you did looks bad." But if they phrased it that way, they'd have to answer the question "Was it actually bad, or not?"
>If someone did something bad, you don't need to talk about "optics." And if they did something that seems bad but that you know isn't, why are you criticizing it at all? You should instead be explaining why it's not as bad as it seems.
https://nitter.net/paulg/status/1728015636771504300#m
The situation is similar if you criticize an action on the grounds that "it upset people". If the action was actually bad, you should be able to explain why directly.
Here are some quotes from the OpenAI charter:
>We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.
>...
>Our primary fiduciary duty is to humanity.
>...
>We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project.
And here's a quote from the OP:
>In October, Toner, who is director of strategy at a think tank in Washington, D.C., co-wrote a paper on AI safety. The paper said OpenAI’s launch of ChatGPT sparked a “sense of urgency inside major tech companies” that led them to fast-track AI products to keep up. It also said Anthropic, an OpenAI competitor, avoided “stoking the flames of AI hype” by waiting to release its chatbot.
>After publication, Altman confronted Toner, saying she had harmed OpenAI by criticizing the company so publicly. Then he went behind her back, people familiar with the situation said.
Who is doing a better job of upholding the charter in this situation: Altman, or Toner?
The "people are upset" criticism essentially states that people became upset when Toner attempted to actually uphold the charter. To me, that says more about those people than it says about Toner.
For this to work you need all layers of society to be represented and since this is intended to be global you'd be looking at the UN. Of course that's 100% anathema to anybody involved with OpenAI or even any cross section of HN because 'bureaucrats don't get technology and the UN can't agree on anything'. But if it really is what it seems to be that to me seems to be the only venue where this can be addressed.
Of course the race is on and the first party to cross the threshold will get the gold and never mind the consequences so I expect any attempt at oversight to be murdered the second they have it working.
Somewhere I got the notion that the UN is generally considered to be dysfunctional these days, and that's part of the reason there's been a shift in diplomacy to other venues like the G20. Representativeness is important, but it's just one of several important desiderata in AI governance.
It seems to me that the current important decisions facing OpenAI are for the most part fairly technical, so it's good for their oversight to be technical. Representativeness becomes more important later. I'd say if OpenAI actually does succeed in developing AGI, that's the point at which the board should be replacing itself with some other, more representative institution.
>Of course the race is on and the first party to cross the threshold will get the gold and never mind the consequences so I expect any attempt at oversight to be murdered the second they have it working.
Maybe a good question from a governance perspective would be, how can we configure the institutional incentives to discourage this, or at least select for leaders with strong character and ethics who won't ruthlessly seek power at all costs?
I've been criticizing Sam Altman a fair amount lately, but it sounds like you take it much farther in essentially viewing him as some sort of crime boss, or else you're assuming that anyone with significant power will automatically adopt crime boss ethics as a result. I doubt the latter statement is true -- for example, I don't believe that Obama adopted crime boss ethics when he became POTUS.
> It seems to me that the current important decisions facing OpenAI are for the most part fairly technical
For OpenAI, yes. But for society the stakes are much higher and non-technical in nature. For instance: mass disruption of the labour market. Note that when we had the industrial revolution the world population was a fraction of what it is today.
> so it's good for their oversight to be technical
But that could change in a heartbeat and by then it is too late. Winner takes all.
> Representativeness becomes more important later.
There won't be a 'later', whoever gets this first will instantly weaponize it and seize power, though you won't know about it until you realize that it's a done deal. Stable dictatorships are one of the more likely outcomes of AI.
> I'd say if OpenAI actually does succeed in developing AGI, that's the point at which the board should be replacing itself with some other, more representative institution.
Right now our best hope is that they can't. If not, all bets are off and you may want to brush up on your Charles Stross.
> Maybe a good question from a governance perspective would be, how can we configure the institutional incentives to discourage this, or at least select for leaders with strong character and ethics who won't ruthlessly seek power at all costs?
Leaders with strong power and those who ruthlessly seek power at all costs have strong overlap, assuming they are not congruent.
> I've been criticizing Sam Altman a fair amount lately, but it sounds like you take it much farther in essentially viewing him as some sort of crime boss, or else you're assuming that anyone with significant power will automatically adopt crime boss ethics as a result.
I see him just like I see Elon Musk: brilliant but dangerous, not necessarily a crime boss, more like someone who doesn't realize that they don't have the right to make the decisions they make and too little in terms of reflective capabilities and humility.
> I doubt the latter statement is true -- for example, I don't believe that Obama adopted crime boss ethics when he became POTUS.
Obama was a huge disappointment, so no argument there, but the flipside is that Trump was exactly what he promised to be and more and the world came out for the worse anyway. It makes you wonder how much worse that can get. Probably a lot worse. I probably don't want to know.
You're characterizing this like it was just a bunch of Twitter armchair quarterbacks.
Which is was.... Plus 90%+ of the boots on the ground who voted no confidence in their leaders.
Only the most obstenate can cling to the belief they are a good leader/communicator with that statistic.
EDIT: There is a feeling that companies are full of unremarkable, out-of-touch, over-compensated leaders who ignore the opinions of their subordinates.
And yet when that situation actually materializes, or mostly materializes, you read opinions like this.
I'm concerned that boots on the ground may be susceptible to motivated cognition due to their financial interest. I suspect that their finances come before upholding the charter for many of them. Why else did they get so upset when Helen said that shutting down OpenAI could be the right thing to do? That's pretty much implied by the charter.
Disqualifying their opinion on OpenAPI leadership seems unjust.
"Altman approached other board members, trying to convince each to fire Toner. Later, some board members swapped notes on their individual discussions with Altman. The group concluded that in one discussion with a board member, Altman left a misleading perception that another member thought Toner should leave, the people said. By this point, several of OpenAI’s then-directors already had concerns about Altman’s honesty, people familiar with their thinking said."
Leadership is fundamentally about manipulating people. If the people already agreed, they wouldn't need leadership. Your job as leader is to get them all working in more or less the same direction, even though they don't really want to.
Honesty is a great way to do that, when it works. Honesty has the virtue of not having to keep track of your lies, and you don't get caught in an inconsistency.
But it doesn't always work, and then you have to consider alternatives. Sometimes people are dumb. Sometimes people are wrong. Sometimes people are hostile. Sometimes people are right, but in a way that can't actually be implemented, and you need to go with a sub-optimal path despite that.
At core, you can't lead if you don't believe that you've listened to all of the concerns and reached the solution that is best for the group as a whole, according to whatever definition of "best" and "whole" you were hired for. Sometimes that's going to force you to use other means that just telling people the truth and hoping they reach the same conclusion.
Leadership is fundamentally a people problem, not a technical one. And sometimes that's going to mean being dishonest. A good leader does it as little as possible, but simply dismissing it as a tactic is a recipe for failure because it hides a fundamental truth about why and how people behave in groups.
Which successful CEO takes direction from a board that meets a few times a year?
"Managing the board" is one of the major jobs of the CEO, most of that work done outside those key moments when the board meets to formally vote on what they usually already decided based on earlier discussions between themselves, CEO, shareholders and others.
Honestly? Nearly all of them. It's how the whole thing is supposed to work, and for the most part it works ok. There are exceptions.
Hard disagree. A leader cultivates agreement and cooperation through clear articulation of a vision and a path to get there.
lying to board is dick move, manoeuvring to get someone fired is a dick move. If what we read is mostly true, then I can see why the board fired him, after all, how can they trust him?
Not that that's actually ever going to be the case at OpenAI again, but the parent has the chain of command here totally bass-ackwards.
In theory, yes. In practice: Not always.
In truth it's both. That said, leadership doesn't mean lying. Particularly not about what another person has said. Very particularly not when it's trivial for them to compare notes.
Your line of reasoning fulfills the stereotype that tech people are sorely lacking in a well-rounded education.
This is wrong. Manipulating people is not leadership. It might be management in some circumstances, but management is not leadership.
> If the people already agreed, they wouldn't need leadership.
This is wrong too. People can agree on a goal without seeing how to get there. Leadership is not just about seeing the goal. It's about leading people there. But if you have to manipulate the people, you're not leading them. You're just using them for your own ends.
This is true...if you are the leader of a cult, which exist to deify you more than benefit the group.
> If the people already agreed, they wouldn't need leadership. Your job as leader is to get them all working in more or less the same direction, even though they don't really want to.
Yes, but if you're having to puppet people to do it, it's not leadership. It's just manipulation. You should be able to do this through inspiration, and transparently in group settings, not by compartmentalizing lies and half-truths privately. That's just fucking embarrassing. Altman's leadership skills are on par with a 14-year old influencer.
I feel like you're taking the argument for why "we let Pearl Harbor happen" and using it to excuse the whims of a psychopath.
The dude essentially threatened to destroy the company if he didn't get his way. Might does not make right. Sam turned out to have more leverage here; that doesn't make him a good person.
>You don’t fire someone so important for that and extrapolate it into equating human extinction.
On the other hand, would you really want a company developing existentially dangerous technology to be lead by someone who's known as a slippery operator?
My hunch is that part of the board's motivation was a feeling that they should have unusually high standards for the CEO of OpenAI.
>...
>Pretty much everyone who works on AI thinks that they're 'one of the good people'. Statistically, this is unlikely to be the case.
>...
>Proactively giving up power is one of the hardest things for people to do. Giving up power and money is even harder. AI orgs are rapidly gathering power and money and it's not clear they have right incentives to willfully shed their own power. This sets us up for racing.
https://threadreaderapp.com/thread/1555980412333133824
>Also who is to say deceleration in the age of AI isn’t worse than acceleration? Every govt entity is developing it, and it could be a national sec issue to speed it up faster than an adversary
See e.g. https://pauseai.info/faq#if-we-pause-how-about-china
Silicon Valley seems to have a fairly strong monopoly on AI talent.
This and the FTX saga just shows EA as nerdy movement in arrested development.
- As PG pointed out Sam would find his way to the top of an island of cannibals. If you're generous to Toner's side in this article she likely _was_ alarmed by how Sam tried to get other board members to turn against her. When she walked into the arena with Altman to try and remove him however he had much more experience and orchestrated an incredible counter-coup.
- There must be an army of attorneys representing the board members, and investors because there are huge potential legal pitfalls to destroying value, or failing fiduciary duties.
- We don't know a lot because anything put out in public has to be spotless legally. If you're a board member and you know the truth, the only upside to leaking more info is clout, but the downsides are being sued and tied up in court losing a lot of time and money.
- People seem dissatisfied with what was revealed in this article but Toner has likely been advised by lots of counsel on exactly what can be said, and the list of those things is probably miniscule. I'd say she's doing the best she can to give the public more info and I'm happy she's taken the effort to do so.
Nit: Altman may have orchestrated a coup, but the board certainly did not. The board is the legitimate authority, and in no way is a board firing its subordinate a "coup." Rather, it's executing its legitimate power.
Coups triggered when subordinate feels threatened is a pretty common pattern. IIRC, the recent coup in Niger was triggered when the leader of the presidential guard thought the president was about to fire him, too power in a coup rather than lose his job.
The "fat lady hasn't sung" point works both ways -- for all we know, the new board's investigation into this incident will be incredibly damning for Sam.
Sam was on the board. They didn't fire a subordinate they kicked a peer out of the group.
Why did they take out the chairman of the board.
> advised by lots of counsel on exactly what can be said, and the list of those things is probably miniscule.
Hmmm, in the US yes and no. First, US lawyers point out risk in every direction. Because nothing is risk-less and the law is unclear or unsettled (and anyone can sue for just about anything). Once they have duly warned to their satisfaction however, their job becomes to come up with a theory on how to do exactly what it is you want to do. The two are different tasks, and are not incompatible.
That's basically the definition of a good business lawyer.
Companies implode all the time. Sure you might get a lawsuit but the company would pay it or the directors insurance would.
HN is always scared about lawsuits. If Trump taught us anything is that there’s a lot of leeway in that area.
> Toner graduated from the University of Melbourne, Australia, in 2014 with a degree in chemical engineering [...] In 2019, she spent nine months in Beijing studying its AI ecosystem. When she returned, Toner helped establish a research organization at Georgetown University, called the Center for Security and Emerging Technology, [...] She succeeded her former manager from Open Philanthropy, Holden Karnofsky, on the OpenAI board in 2021 after he stepped down.
Honest question, how do people find themselves in the board of one of the hottest startups at the tender age of 31? Are they geniuses or is it all about connections?
Her brothers: https://www.vesparum.com/origins
So the short answer is wealth and connections, or daddy.
Going fwd, I wonder why they cannot convert the structure to a traditional c-corp. Supposedly: “tax issues”.
Whatever problems OpenAI dealt with last week, the current non-profit structure will continue to cause future problems IMO.
Great learning opportunity for those that have not taken time to learn and understand non-profit structures in the USA.
this is clearly an opportunity for the uninformed to study the US non-profit system, unique in the world. For example, I recall a delegation from Japan touring the HQ of a non-profit educational group in San Francisco. Like the p-poster here, they apparently had no clue why or how capable people could come together and form stable work culture aside from "hierarchical, profit motive in USD$" (.. and people did actual work, unlike many management level stock manipulators).
[0] https://en.wikipedia.org/wiki/International_Baccalaureate
"In the aftermath of the OpenAI governance battle, the narrative has been dominated by the idea that “the capitalists won.” While this conclusion is catchy, it misses the point.
The real (and bigger) story is: The founders of OpenAI set out to create a nonprofit public institution to ensure that AI’s evolution aligns with the interests of humanity at large. Last month, it became clear that they failed.
What is also clear: Public institutions that prioritize the public good over short-term profits are desperately needed for the AI era."
I read it as both. If OpenAI collapses operationally then the non profit fails at its mission too. That’s the board she is on and that’s the fiduciary duty relevant
(She may also be on the for profit board idk)
Not necessarily true.
"We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome [...] if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project."
Because, IIRC, when a charity nonprofit like OpenAI, Inc.-- a 501(c)(3), other nonprofits are different -- converts to a for-profit (C-corp or otherwise doesn't matter), all the assets acquired while it was a charity must either remain dedicated to and used exclusively for the charitable purpose, be returned, or be donated to charity (or sold, and then the proceeds treated the same way.) For the charity OpenAI, Inc., this would include its interest in OpenAI, LLC (the wholly owned subsidiary through which it exercises control of OpenAI, LP and OpenAI Global, LLC -- the latter being the for profit entity that actually sells product, etc.), and OpenAI, LP (the holding company which in turn is majority owner of OpenAI Global, LLC).
also OpenAI Inc. for profit doesnt exist, a wholly owned LLC subsidiary does. so they would need to divest of that LLC first (by selling the shares)
the way around the self dealing regulations are to reduce their ownership below 20% - 35% (yes, its that range, for multiple reasons)
good thing here is that most venture backed companies subvert this already by simply share dilution and youre breaking into one the best secrets of Silicon Valley. Non-profits pump the value of startups and launder the reputation of the founders because their ideas get so much “validation” so quickly. (non profits they partially control act as lead investors for startups they control. they both sell shares at higher price, ideally: founders get QSBS tax free treatment, non profit retains tax free capital gains too. nothing wrong with that, you just probably cant do it too)
so open ai for profit llc needs to get rid of the PPU’s and bring in outside investors to dilute ownership enough, but two birds one stone: the outside investors have to get a higher valuation
The for-profit "OpenAI" (OpenAI Global LLC) is not a wholly-owned subsidiary of anything, it is majority owned by a holding company (OpenAI, LP), which in turn is majority owned by the OpenAI charity (OpenAI, Inc.)
There is a wholly owned LLC subsidiary of the charity (OpenAI GP, LLC), but that is the entity through which OpenAI, Inc., exercises control of OpenAI, LP and OpenAI Global, LLC (why it doesn't do so directly for OpenAI, LP, and either directly or via OpenAI, LP, for OpenAI Global, LLC, don't ask me), it is not the for-profit.
Both OpenAI, LP, and OpenAI Global, LLC have other investors (Microsoft being a big one for OpenAI Globla, LLC) already.
one explanation is “blockers”. basically regulations imposed on one kind of entity such as the non profit, fail when there is one degree of separation via a “blocker” entity.
LP’s can have undisclosed silent limited partners, intentionally less flexible than an LLC which can do the same
Fiduciary doesn't mean financial or monetary duty, it means holding something in trust. Violating a fiduciary duty could be financial, social, reputational, or in this case, literally destroying the operation of the company you were entrusted with.
Yeah, that smells like a bullshit threat. Anyone would do well to ignore it.
The folks who argue that it is a CEO's duty to serve the board do not understand governance or power.
Open AI would have wallowed in irrelevance had Altman not raised the billions from Microsoft to fund its research. Because he was able to do that, and built the relationship with Nadella, he had and has power. Toner's behavior seems naive in that context. But naiveté among academics does not surprise me.
You can be fairly certain that the board will be stocked with 'yes-men' (and women) beholden to Altman. He will not make that same mistake twice.
"Pournelle's Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: those who work to further the actual goals of the organization, and those who work for the organization itself. Examples in education would be teachers who work and sacrifice to teach children, vs. union representative who work to protect any teacher including the most incompetent. The Iron Law states that in all cases, the second type of person will always gain control of the organization, and will always write the rules under which the organization functions."
> organization she pledged to serve
First there is no pledge, second I think she served the Open AI Charter.
I'd say so, and in fact it's older than you might think.
Here's a lit review from 2018 for example: https://arxiv.org/pdf/1805.01109.pdf
This post is more recent, but understanding it might require some context: https://www.alignmentforum.org/posts/zaaGsFBeDTpCsYHef/shall...
A group from Cambridge has been running an online course you can take if you're interested in getting up to speed: https://aisafetyfundamentals.com/ Application deadline is early February. I took the course a couple years ago; happy to answer any questions.
Nobody involved is being transparent about what went down here. Not sure we'll find out the full story here...
I think this Reddit comment captures the most plausible story at this point: https://www.reddit.com/r/MachineLearning/comments/1812w04/co...
That's not a good look on Altman at all, it looks like Altman was using divide-and-conquer in a skeezy way, spreading around negative impression, even rumors, about his bosses, as a means dispose of a problem.
Just like “terrorism” and “drugs” and all the other vaguely menacing things that ended up being worth a pile of money to some incumbent interest, the real story will be boring: some crooks used a fancy-ass LLM to take a bank apart or something. The bank calls the cops, this happens about twice before the cops have a division for this sort of thing, and sooner or later someone is in a courtroom.
It would be nauseatingly boring to get into how many times this has happened.
Also predictably depressing, the real story, which is yet another brick in the wall of “we’re handing the reigns to people who make movie Zuck look flat fucking normal”, goes largely unreported.
We used to complain about dual-class share structures.
Sam just eviscerated a board of an ostensible charity with the full backing of MSFT and got himself installed as Ungovernable God Emperor of Arrakis for life. Still doing the eyeball scanner thing.
Still has never shown the public he can handle numpy to the tune of MNIST.
This is all in broad daylight and the Gemini launch gets clobbered because the 4-series comparable one comes out in January.
Sam’s a smart guy, I’m quite sure he’d pay attention if the braintrust (this site) complained loudly.
But the message is: “more, not less”.
> Toner declined to provide specific details on why she and the three others voted to fire Altman from OpenAI
Does she see herself as more trustworthy? She can't even be bothered to give an excuse for the firing.
Someone like this couldn't be trusted in literally any function, how did she get a board seat on OpenAI?
She's the leading academic on the issue of AI safety. It's really ridiculous people don't even know her name and say random things about her, not realizing she's a rock star in her field.
Her leaving the board is a tremendous loss for OpenAI, about as terrible as Ilya leaving the board. These are two giants of AI currently.
Sounds like a made up position to me. Just another bureaucrat with a cool sounding name
https://scholar.google.com/citations?user=NNnQg0MAAAAJ&hl=en
Like, it's not a bad publication record, especially for a junior researcher (heck, her first participation in other people's papers is just in 2018 and her own work starts in 2021-2022, so she's effectively just getting started and may have many opportunities to prove herself in the future), it does indicate doing a couple years of research and might qualify for an okayish faculty job when (if?) she gets her PhD, but it's not also something indicating a senior researcher that's the pillar of anything and has had some impact - a grad student who's lucky to study in a strong leading researcher's lab and does some work on two of the advisor's papers might have a similar publication record already at their graduation before their career has even seriously started.
Many of her co-authors have significantly larger research impact and personal research (https://scholar.google.com/citations?hl=en&user=VclFrJ8AAAAJ or https://scholar.google.com/citations?hl=en&user=MbBntPgAAAAJ for example or https://scholar.google.com/citations?hl=en&user=0-G2eiEAAAAJ) - which also raises a question of how much each co-author did in a two-author paper where the leading author is a strong researcher and the second author is a "Director of Strategy and Foundational Research Grants" i.e. getting money for that research, so labeling her as a leading researcher (much less the leading researcher) of some field seems misleading as the publication record would indicate that she's not even on the same level as they are.
https://scholar.google.com/citations?hl=en&user=6-e-ZBEAAAAJ
It sounds like someone who talked a lot but didn’t actually create anything, change anything, or make anyone’s lives better.
Happy to revise my perspective if someone can provide any concrete data on her impact.
She was on the board that fired Sam Altman, that had the courage to fire Sam Altman, presumably (according to the article at least) because he is a liar and she and three other board members did not trust him to be loyal to the non-profit's mission of working to keep AI safe.
"In 2019, she spent nine months in Beijing studying its AI ecosystem. When she returned, Toner helped establish a research organization at Georgetown University, called the Center for Security and Emerging Technology, where she continues to work."
I don't think "rock star" is quite the right term.
Unless you think David Geffen or Lester Bangs are rock stars. Normally rock stars sing or play an instrument. Someone who got a nice title in a nonprofit by writing about rock is not a rock star.
Yann LeCun is a leading academic...
I'm embarrassed she comes from my country, but not surprised about the particular uni she attended within my country.
I'm doubtful that the "childish idealist" description is accurate. But even if it was, given the choice between having existentially dangerous AI developed by a childish idealist vs a slippery machiavellian like Sam, I would choose the childish idealist.
The big risk with a childish idealist is that they become the pawn of a slippery machiavellian. Choosing the slippery machiavellian just seems like speedrunning failure.
I’m quite happy with the childish idealist description though. Putting childish idealists in charge of $billions of other people’s money might sound like a great idea to you, but would you put them in charge of your money?
Wikipedia says you have to apply to attend some EA events. That's true for lots of things. TED isn't open to just anyone either. If you want to go to an EA event without applying, most group events are pretty casual and don't require any application: https://forum.effectivealtruism.org/groups (Note: I heard a rumor that EA events in Germany are less likely to be open to the public)
>Putting childish idealists in charge of $billions of other people’s money might sound like a great idea to you, but would you put them in charge of your money?
Well, I have in fact donated significant sums to some EA organizations, and I have a recurring donation to the Good Food Institute (working on meat alternatives). I don't agree with the "childish idealist" description of EA though. From my perspective, it's remarkable how easily Sam Altman seems to manipulate people, and he seems more successful at manipulating people outside of the EA community.
Other than telling me that you have a narrow understanding of a "cult", I'm not sure what this is supposed to communicate.
BTW, for people reading this discussion, here's an alternative explanation of why EA tends to be controversial: https://www.richardhanania.com/p/effective-altruism-thinks-y...
Peter Singer is more of an EA popularizer than an EA leader. The post you linked is the only post he has made in the past ~9 years: https://forum.effectivealtruism.org/users/peter_singer The tweet you linked is related to his Journal of Controversial Ideas, not EA.
It's certainly not the case that Singer should be taken as representative of the average EA, you'd have to look at surveys for that: https://forum.effectivealtruism.org/s/FxFwhFG227F6FgnKk However EA tends to be friendly to contrarian thought experiments (not usually taking it to Peter Singer levels). And EAs usually prefer to focus on getting facts right over condemning each other in an inflammatory way.
Is EA distancing itself from Singer (who's views are, by all accounts, extremely influential in the EA community), as they did with SBF (whose hedge fund's initial $170m seed capital was raised from EA devotees)? When multiple high-profile people associated with EA are proven to be....problematic, should we pretend the common factor linking them is insignificant?
Inflammatory? I thought I was quite restrained compared to some of the commenters on that Twitter thread.
From article, this is what I know about her:
>"Toner graduated from the University of Melbourne, Australia, in 2014 with a degree in chemical engineering and subsequently worked as a research analyst at a series of firms, including Open Philanthropy, a foundation that makes grants based on the effective-altruism philosophy."
>"In 2019, she spent nine months in Beijing studying its AI ecosystem. When she returned, Toner helped establish a research organization at Georgetown University, called the Center for Security and Emerging Technology, where she continues to work."
She's been working in the field of AI safety for what 3 years at most? I find it very hard to believe that she's "thee" leading academic on the issue of safety. Stuart Russell's group, the Center for Human Compatible AI at UC Berkeley, MIT's The Algorithmic Alignment Group and David Kruegger's group at Cambridge are all academic leaders and groups with great expertise in this subdomain.
Further it's not even clear what "studying Beijing's AI ecosystem" actually means. Does this refer to the city? The university? The CCPs' surveillance state(not likely)? It's very hand wavy.
I don't have any problem with what she did or her views. I also think it's a loss for the company. I do question her bona fides for being called the leading academic on the issue though.
I agree that outsiders can't weigh in on this topic without statements from the board.
But as the (former) board member of a private organization, she's not accountable to the public. Making statements about why she fired Altman can carry substantial legal risk.
Exactly. Being a rent-seeking grifter doesn't make you an expert. In anything.
I'm amazed anyone can side with Khosla after he blatantly showed his $ > ethics side when it came to the Martin's Beach scandal[0]. In a battle of $ and ethics, finding yourself agreeing with Khosla is a smoking gun where someone stands.
That we have VC's character assassinating someone in contrast to all the people actually around her who can vouch, is quite a sad legacy.
[0] https://www.washingtonpost.com/technology/2020/01/08/califor...
Those who can, do. Those who can't, write about it.
I was there, as an engineer. I'm a character in the novel.
https://scholar.google.com/citations?user=NNnQg0MAAAAJ&hl=en
None of them are about chemical engineering, which has nothing much to do with AI, anyway, so I don't know why you mention that.
when did she go over to the Dark Side?
https://scholar.google.com/citations?user=NNnQg0MAAAAJ&hl=en
"bridge the AI communities of the US and China": what does that mean? She organized some meetings where platitudes were intoned?
She replaced Holden Karnofsky as a representative of the "effective altruist" group, Open Philanthropy, who donated money early on to OpenAI.
From the article:
"She succeeded her former manager from Open Philanthropy, Holden Karnofsky, on the OpenAI board in 2021 after he stepped down. His wife co-founded OpenAI rival Anthropic."
I can almost promise you there is a lawyer telling her not to talk about the firing.
Some people just fail upwards. Mind boggling.
Was he fired or he stepped down to be the CEO of Open AI?
> Some people just fail upwards. Mind boggling.
Didnt he hire the initial team and get investor money, that is a pretty good skill.
Brockman hired the top researchers with help from Yoshua Bengio.
Elon Musk was the biggest initial donor, and he is on record saying that Ilya was the key hire.
Well, if that was the case it actually was fine but there is no hard proof that it was like that and if it was they messed it up. So failure on all counts.
> “Fancy titles like ‘Director of Strategy at Georgetown’s Center for Security and Emerging Technology’ can lead to a false sense of understanding of the complex process of entrepreneurial innovation,” Khosla wrote in an essay in tech-news publication the Information, referring to Toner and her current position.
Strong words from the infamous beach villain and resident asshole, Vinod Khosla...
Every Generation Gets the Beach Villain It Deserves (NYT) https://archive.is/6mss9
This is the biggest news of it all So all the Q* thing and Ilya's safety position and other theories were incorrect?
One point here is that there's no licensing requirement for AI engineers like you see in some other fields of engineering. As someone recently quipped on Twitter:
> did you know that there are currently no licensing requirements in the US for training potentially world-threatening AI models, but there are in the state of Louisiana for selling flower arrangements
https://nitter.net/littIeramblings/status/173092523732656542...
In any case, what's been happening in AI engineering is significantly different than in other fields of engineering.
There is no consensus as to the magnitude of AI risk, although extinction probability estimates are generally high: https://nitter.net/AISafetyMemes/status/1729892336782524676#...
In practice, AI engineers self-select into different places of employment depending on how worried they are. If you're not very worried, you work at Meta. If you're significantly worried, you work at Anthropic or even someplace like MIRI. Most AI labs have a safety culture that's somewhere in between those extremes. [Obviously, from the point of view of preventing extinction, this situation seems unsatisfactory.]
With regard to the point about the board -- their job was to supervise the CEO, and they say Sam was making that difficult. See https://nitter.net/hlntnr/status/1730034020140912901#m
It's more of a smear. If you're the one arguing from an AI safety standpoint, it means your opponent is not being safe and is dangerous.
You should be expected to articulate why your position is "safer" and how "safe" is defined.
In my opinion most AI safety arguments are about vaguely defined variables like speed: "it's going too fast" - to who? who defines what fast is? do you slow down 2x or 10x? do competitors and/or rival nations less concerned zip by and the end result is still the same?
There is some validity to the discussion if AI is being "racist" or "biased" (again however you define these) but again competitors and/or rival nations may be less concerned and the end result may still be the same.
If anything the "safest" option is to define what your concerns are, then race ahead to try to be the canonical solution or defined standard in order to set standards for the rest to follow or be bound by.
Otherwise I don't see what this means now; that the LLMs e.g. dont use racist terms? ok, great, nice, but how is that anything more than what you need to do on the web now anyway? how's that related to "AI" at all?
What I'd love to see is that this gambit backfires and instead we start talking about "tech safety" and that creates actual regulations with teeth that cut down the techzillas a bit (or a lot).
I respect that she and other board members didn’t like that Sam was trying to manipulate them into ousting her but their counterattack was poorly thought out.
https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...
I have worked for many CEOs, and (surprise!) fewer than 100% of them are sociopaths. Good grief.
I am not saying this necessarily as a criticism, but self-delusion appears to be an important part of many founder and executive stories.
Insightful.
Somebody put her there, told her "you know, say it's for humanity or something" and she actually believed it.
No, it's for people to get rich.
But you seem to have something in mind. Tell me.
So there are many particulars that mean pattern matching to a standard board dispute will lose something. I think it's likely many of the primary actors have, at various times, had non-strictly-pecuniary motives. That one side won doesn't mean that the other side was always a farce.