Imnimo

The bad boy of science!

My read is not so much "if we say this is dangerously powerful, it will make people want to buy our product", but rather that there is a significant segment of AI researchers for whom x-risk, AI alignment, etc. is a deal-breaker issue. And so the Sam Altmans of the world have to treat these concerns as serious to attract and retain talent. See for example OpenAI's pledge to dedicate 20% of their compute to safety research. I don't get the sense that Sam ever intended to follow through on that, but it was very important to a segment of his employees. And it seems like trying to play both sides of this at least contributed to Ilya's departure.

On the other hand, it seems like Dario is himself a bit more of a true believer.

Unsurprising from Google, but still bad. If Google has no right to object to a particular use, this is equivalent in practice to "any use, lawful or not".

Back when Arena was first announced, there was an interesting line in their write-up:

https://magic.wizards.com/en/news/feature/everything-you-nee...

>We've created an all-new Games Rules Engine (GRE) that uses sophisticated machine learning that can read any card we can dream up for Magic. That means the shackles are off for our industry-leading designers to build and create cards and in-depth gameplay around new mechanics and unexpected but widly fun concepts, all of which can be adapted for MTG Arena thanks to the new GRE under the hood.

At the time, this claim of using "sophisticated machine learning" to (apparently?) translate natural language card text into code that a rules engine could enforce struck me as obviously fake. Now nearly ten years later, AI is starting to reach a level where this is plausible.

In their letter, the union writes:

>Over the past few years, pressure has ramped up from leadership to adopt LLMs and Gen AI tools in various aspects of our work at WOTC, often over the explicit concerns of impacted employees

I'm curious if this would include fighting against turning WotC's old fanciful claim into a reality as the technology matures?

Why should we think that pro-social capabilities are simply not expressible by weight-based ANN architectures?

>Tigers, hippos and SARS-CoV-2 also developed ”through evolution”. That does not make them safe to work around.

Right, but the article seems to argue that there is some important distinction between natural brains and trained LLMs with respect to "niceness":

>OpenAI has enormous teams of people who spend time talking to LLMs, evaluating what they say, and adjusting weights to make them nice. They also build secondary LLMs which double-check that the core LLM is not telling people how to build pipe bombs. Both of these things are optional and expensive. All it takes to get an unaligned model is for an unscrupulous entity to train one and not do that work—or to do it poorly.

As you point out, nature offers no more of a guarantee here. There is nothing magical about evolution that promises to produce things that are nice to humans. Natural human niceness is a product of the optimization objectives of evolution, just as LLM niceness is a product of the training objectives and data. If the author believes that evolution was able to produce something robustly "nice", there's good reason to believe the same can be achieved by gradient descent.

>Unlike human brains, which are biologically predisposed to acquire prosocial behavior, there is nothing intrinsic in the mathematics or hardware that ensures models are nice.

How did brains acquire this predisposition if there is nothing intrinsic in the mathematics or hardware? The answer is "through evolution" which is just an alternative optimization procedure.

It's interesting to me that OpenAI considers scraping to be a form of abuse.

So, if you look at the way the scoring works, 100% is the max. For each task, you get full credit if you solve in a number of steps less than or equal to the baseline. If you solve it with more steps, you get points off. But each task is scored independently, and you can't "make up" for solving one slowly by solving another quickly.

Like suppose there were only two tasks, each with a baseline score of solving in 100 steps. You come along and you solve one in only 50 steps, and the other in 200 steps. You might hope that since you solved one twice as quickly as the baseline, but the other twice as slowly, those would balance out and you'd get full credit. Instead, your scores are 1.0 for the first task, and 0.25 (scoring is quadratic) for the second task, and your total benchmark score is a mere 0.625.

Suppose you construct a Mechanical Turk AI who plays ARC-AGI-3 by, for each task, randomly selecting one of the human players who attempted it, and scoring them as an AI taking those same actions would be scored. What score does this Turk get? It must be <100% since sometimes the random human will take more steps than the second best, but without knowing whether it's 90% or 50% it's very hard for me to contextualize AI scores on this benchmark.

It was neat to be able to try my own prompts and get a sense of what the state of video generation was. But I certainly never generated something that I thought I got real value out of on its own merits, and I still don't understand why there was a social media component to the app.

The fact that the LLM appears to never assign an actual 0 or 10 makes me suspicious. Especially when the prompt includes explicit examples of what counts as a 10.

I think the problem for xAI is that it can really only hire two types of researchers - people who are philosophically aligned with Elon, and people who are solely money-motivated (not a judgment). But frontier AI research is a field with a lot of top talent who have strong philosophical motivation for their work, and those philosophies are often completely at odds with Elon. OpenAI and Anthropic have philosophical niches that are much better at attracting the current cream of the crop, and I don't really see how xAI can compete with that.

>What we’re learning from this episode is that the government actually has way more leverage over private companies than we realized.

Who is learning this for the first time only now? Even just restricting ourselves to the current administration, look at how many times Trump has directed punitive actions against private entities! Look at his actions against law firms like Perkins Coie or Covington & Burling. This is not something that just arose out of nowhere with Anthropic.

My metric for this kind of stuff is: Did Glaze build the Glaze app?

If someone tells you they're going to handle US communications in a way that is consistent with the FISA Act, that is not a good thing.

None of those explanations are compatible with the pledge of solidarity in the We Will Not Be Divided letter.

I don't see how OpenAI employees who have signed the We Will Not Be Divided letter can continue their employment there in light of this. Surely if OpenAI had insisted upon the same things that Anthropic had, the government would not have signed this agreement. The only plausible explanation is that there is an understanding that OpenAI will not, in practice, enforce the red lines.

Do you have to prove that your 3D printer cannot print a 3D printer which can print a gun?

Ha! I misread it as "Haiku Warrior" and so didn't make the connection. That makes a lot more sense!

Apparently Haiku is a very anxious model.

>The anxiety creeps in: What if they have removal? Should I really commit this early?

>However, anxiety kicks in: What if they have instant-speed removal or a combat trick?

It's also interesting that it doesn't seem to be able to understand why things are happening. It attacks with Gran-Gran (attacking taps the creature), which says, "Whenever Gran-Gran becomes tapped, draw a card, then discard a card." Its next thought is:

>Interesting — there's an "Ability" on the stack asking me to select a card to discard. This must be from one of the opponent's cards. Looking at their graveyard, they played Spider-Sense and Abandon Attachments. The Ability might be from something else or a triggered ability.

Well, typically for higher progressive taxes. Tariffs are typically a regressive tax.

>The randomly selected applicants

Why would you want to randomly select here?

I am in the "did not read a single book" bucket, and have been for many years. I just don't like reading books, and never have.

>An advertising-based business model would introduce incentives that could work against this principle.

I agree with this - I'm not so much worried that ChatGPT is going to silently insert advertising copy into model answers. I'm worried that advertising alongside answers creates bad incentives that then drive future model development. We saw Google Search go down this path.

>By contrast, intrinsic mortality stems from processes originating within the body, including genetic mutations, age-related diseases, and the decline of physiological function with age

So we put genetic diseases in the bucket of intrinsic mortality and then found that intrinsic mortality has a heritable component?

In this sentence, Anthropic makes clear that "be hurtful" and "lead to public embarrassment" are separate and distinct. Otherwise it would not be necessary to specify both. I don't think this is the signal they should be sending the model.

I am somewhat surprised that the constitution includes points to the effect of "don't do stuff that would embarrass Anthropic". That seems like a deviation from Anthropic's views about what constitutes model alignment and safety. Anthropic's research has shown that this sort of training leaks across contexts (e.g. a model trained to write bugs in code will also adopt an "evil" persona elsewhere). I would have expected Anthropic to go out of its way to avoid inducing the model to scheme about PR appearances when formulating its answers.

Yeah, both directly and indirectly. Over time, "sponsored links" became more and more visually indistinguishable form organic results, and advertising incentives drove changes to the search algorithm.

>ChatGPT’s responses will not be influenced by ads

I don't see why I should believe this.