Comment by reissbaker | Hacker News Reader

Comment by reissbaker | Hacker News Reader

Parent Full thread

Three points:

1. I don't think Ilya is equivalent to Carmack in this case — he's been focused on safety and alignment research, not building GPT-[n]. By most accounts Greg Brockman, who quit in disgust over the move, was more impactful than Ilya in recent years, as well as the senior researchers who quit yesterday.

2. I think you are underselling what happened with id: while they didn't blow up as fantastically as Ion Storm (Romero's subsequent company), they slowly faded in prominence, and while graphically advanced, their games no longer represented the pinnacles of innovation that early Carmack+Romero id games represented. They eventually got bought out by Zenimax. Carmack alone was much better than Romero alone, but seemingly not as good as the two combined.

3. I don't think Sam Altman is equivalent to John Romero; Romero's biggest issue at Ion Storm was struggling to ship anything instead of endlessly spinning his wheels chasing perfection — for example, the endless Daikatana delays and rewrites. Ilya's primary issue with Altman was he was shipping too fast, not that he was unable to motivate and push his teams to ship impressive products quickly.

I hope Sam and Greg start a new foundational AI company, and if they do, I am extremely excited to see what they ship. TBH, much more excited than I am currently by OpenAI under a more alignment-and-regulation regime that Ilya and Helen seems to want.

Sutskever has shifted to safety and alignment research this year. Previously he was directly in charge of the development of GPT, from GPT-1 on.

Brockman did an entirely different type of work than Sutskever. Brockman's primary focus was on the infrastructure side of things - by all accounts the software he wrote to manage the pre-training, training, etc., is all world-class and a large part of why they were able to be as efficient as they are, but that is not the same thing as being the brains behind the ML portion.

Until I can trust that when I send an AI agent off to do something that I will be successful without me babysitting and watching over it constantly AI won't truly be transformative (since the human bottleneck will remain).

This is one of the core promises of alignment. Without it how can there be trust? While there are probably short term slow downs with an alignment focus, ultimately it is necessary to avoid throwing darts in the dark.

I wouldn't mind a focus on reliably following tasks with greater intelligence; what I think is negative utility is focusing more compute and research resources on hypothetical superintelligence alignment — the entire focus of Ilya's "Superalignment" project — when GPT-4 is still way, way sub-human-intelligence. For example, I don't think the GPT Store was in any way a dangerous idea, which seems to have been Ilya's claimed safety red line.

I wouldn't call GPT-4 sub-human intelligence. While it's intelligence is less robust aggregate human intelligence, I don't think there is any one person alive who can compete with the breadth of GPT-4 knowledge.

I also think that the potential of what currently is possible with existing models has not been fully realized. Good prompting strategies and reflection may already be able to produce a system that is effectively AGI. Might already exist in several labs.

Wikipedia has broader knowledge, and yet no one calls it intelligent. I'm talking about reasoning capability, and GPT-4 is well below human on complex tasks, especially multi-step tasks, which is why "autonomous agents" like AutoGPT, BabyAGI, etc are not yet very useful.

For example: https://arxiv.org/abs/2311.09247

Reply on news.ycombinator.com