FinchNova12

I agree that if you have the weights you can use/train a model with the same architecture, and that you won't get the exact weights on your own due to randomness. But isn't data an extremely important part of your ability to effectively train/finetune? It might be much harder to get close to the level of the open weight model if you don't have the data that made it, which is why I think the open weights vs open source distinction is useful.

I thought there is discussion circulating around whether the main reason Kimi is impressive is due to distillation. Though possibly if this occurred, this was just one component of their training pipeline and not a majority

"Kimi is largely a byproduct of distillation" and "Kimi is introducing new and novel approaches" are not mutually exclusive, and I'm not sure it's clear from the paper how much of the improvement comes from the new approaches. So I wouldn't take the new approaches to be much evidence about whether the distillation attacks occurred.

Hmm, why do you think this is true? One reason I'm skeptical of this is because RL envs are often purchased (and are not publicly available), and this might be a sizable component of why models are getting better.

They state that they are heavily uncertain:

> We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible.

> Our adversaries are great at espionage, stealing models that fit on a thumb drive is relatively easy, and most tech companies are far from operating in a way that would make this more difficult.

Mostly unrelated to the correctness of the article, but this feels like a bad argument. AFAIK, Anthropic/OpenAI/Google are not having issues with their weights being leaked (are they?). Why is it that Meta's model weights are?

Not sure if you intended this, but it feels like the first sentence of your argument is more broadly a critique of the credentials of AI Safety proponents. Maybe you are distinguishing between doomers vs broader AI Safety proponents, but if not, I feel like the counterargument is that most people on the CAIS letter (https://www.safe.ai/work/statement-on-ai-risk) interface quite frequently with these AI models and are also (purportedly) seriously concerned about AI safety

Previous discussion: https://news.ycombinator.com/item?id=28520221

Okay this is reasonable, thanks for clarifying

What? How is this not saying "Well, it might be in the best interests of humanity for OpenAI to do [hypothetical thing that seems pretty bad that OpenAI has never suggested to do], and because they may consider doing said thing, we shouldn't trust them"?

No, you can blame EAs for making stupid decisions and being generally bad people if they are making stupid decisions and being generally bad people. I'll second that I'm sorry for the bad experiences you've had with EAs.

In my experiences, lots of them are simply good people trying to do more good. I hope you can have better experiences with them in the future.

While I agree that the rhetoric around AI Safety would be better if it tried to address some of the benefits (and not embody the full doomer vibe), I don't think many of the 'core thinkers' are unaware of the benefits in AGI. I don't fully agree with this paper's conclusions, but I think https://nickbostrom.com/astronomical/waste is one piece that embodies this style of thinking well!