Cogito Preview: IDA as a path to general superintelligence

I can somewhat understand people developing AGI, but directly working on superintelligence is on extremely shaky ethical ground. A good proportion of AI researchers and philosophers believe superintelligence stands a significant chance of displacing humanity and it is widely regarded as one of the most, if not the most, dangerous technology yet to be created.

Crazy that this is legal.

> We train LLMs through Iterated Distillation and Amplification - an alignment strategy which is not upper bounded by overseer intelligence. Concretely, each iteration involves the following steps: Step 1 (Amplification) - Creating higher intelligence capabilities via subroutines that usually involve more computation. Step 2 (Distillation) - Distilling the higher intelligence back to the model's parameters to internalize the amplified capability.

I agree with their assessment that achieving superintelligence likely isn't possible when using purely human training data, and that iterative self improvement will be a necessary part of the recipe as well.

However, their strategy leans quite heavily on "amplification" without laying out any concrete mechanism that would allow it to generate super human training data while maintaining generality. Because of that, this strikes me as simply an iterative improvement on LLM training rather than a breakthrough that could lead to AGI. If they can figure out that aspect, then they'll have something much more compelling on their hands.

Either way, having OSS models of this caliber is great. The fact that they work with HF, Ollama, and a few other APIs means you can evaluate if it's better for some purpose quite easily.

I really don’t know what to do with posts like these — a startup announced today supposedly beats all other OSS models out of the gate??

I mean, they have screenshots of tables and stuff so they’re presumably not outright lying, but I’ve been burned before in this space. I guess I’ll just have to wait and see if others start freaking out about them…

Tech-wise the IDA algorithm seems promising! Aiming for “superintelligence” is a little bold considering that counts on an (inherently-unpredictable!) intelligence explosion from a human-level model, but I’m sure it helped them raise at least.

Thanks for posting! Keeping my eye on y’all :)