maizeq

2. I thought rare individuals who did not develop language abilities (e.g due to isolation) still had memories of their time prior to thought. The most obvious example to me is Helen Keller, who writes about her time prior to meeting her teacher.

The point here is that malicious hidden behaviour encoded during pre-training seems to be very resistant to generic finetuning without knowing what the hidden behaviour is.

If random websites start including hidden or discreet bits of text which include malicious instructions, they might be activated post-hoc to get a model to do something nefarious. This impacts open source and closed source models alike since they all general train on trillions of tokens which can’t be manually verified for hidden traps like this.

This is instead a question of whether Sama preferentially made a contract with Rain, perhaps despite better alternatives. (Which would benefit him but might not be in the best interests of stake holders).

It’s possible it was the correct decision though, and his stake is just a fortunate side-effect, after all Sam is also invested in other chip companies (Cerebras I believe?).

Here are some seemingly obvious intuitions for me, which all together add up to the obviousness of the risk here.

(1) Current ML models already exhibit all the hallmarks of successful unsupervised intelligence, they simply need to be scaled up and stabilised. This is clear due to results from: model-based RL (e.g Dreamer), and the emergence of causal factors being learnt by even simple models with no explicit supervision (e.g beta-VAE, interpretability research into neurons of LSTMs, etc, etc). The ability to identify causal factors and their relationships and dynamics without explicit supervision, to me, matches every definition of intelligence once can think of.

(2) Current ML learning algorithms (I.e backprop) suggest significantly more efficient credit assignment than that which is employed by the brain. The best example of this is the amount of knowledge distilled per bit in GPT vs the average human. GPT-3 has 170B parameters (Turbo is suspected to have even less), if each parameter was 4 bytes (an extreme case), this would be 5.4 trillion bits. The brain has ~100 trillion connections, even if each connection is a single bit, this is multiple orders of magnitude more bits than GPT-3. Yet GPT-3 can answer questions on quantum physics, just as well as it can translation medicine, just as well as it can Russian literature, etc. etc. This suggests that the idea we will be outpaced is already not a question of how, but of when.

(3) Large intelligent systems will be used for things other than just knowledge extraction. This is perhaps the most key element of this. EVEN if intelligent systems are not programmed - or accidentally embedded with, as in LLMs - with self-motivating or agentic behaviour, we will use them in such a way. That is to say, at some point, we will ask these intelligent systems to “do things”, I.e act upon the world according to our intentions.

(4) Lastly, superintelligent systems that are asked to “do something”, will inevitably do something we do not actually desire. Some researchers, like Yann LeCunn, object to this last bit and believe that we can simply tell them not to do these things. But this misses the fact that even the slightest mis-alignment between our intentions and an AIs could result in catastrophe very rapidly just based on the speed at which a super-intelligence can operate. The most clear cut case of this was the early days of “Sydney”, the Bing AI powered by ChatGPT before it was completely aligned. At one point Sydney was threatening its users, asking them to apologise to it, and going haywire. At the level of a simple chat bot, this is merely a cute local minima the AI has gotten stuck in. At the level of a super-intelligence, the results could be far worse.

https://www.theverge.com/2023/11/18/23967199/breaking-openai...

“The OpenAI board is in discussions with Sam Altman to return to the company as its CEO, according to multiple people familiar with the matter. One of them said Altman, who was suddenly fired by the board on Friday with no notice, is “ambivalent” about coming back and would want significant governance changes.

Update November 18th, 5:35PM PT: A source close to Altman says the board had agreed in principle to resign and to allow Altman and Brockman to return, but has since waffled”

Do you want me to hold your hand to google anything else or is that enough?

Former-chairman*, he was removed from the board immediately after Sam’s ouster. OpenAI has 700 employees, 4 resignations do not make an exodus.

So many articles with no real sources saying the board was desperate for him to come back, or reneged on their decision, or that there would be an exodus of employees!

I’m glad the board wasn’t browbeaten by Sam and his cohort of VC friends.

I am also incredibly doubtful this will have any meaningful impact in terms of engineers/researchers who will choose to leave. Most of the core group of researchers and engineers joined OpenAI the non-profit, not OpenAI the LLC, that is to say I assume many have strong feelings about safety around AI, which judging by the employee testimonies in the recent Atlantic article Sam acted negligently towards.

Almost every researcher I have spoken believes that real risk exists, to some degree or other. Recent surveys of people in industry have largely borne this out - your anecdote sounds more like an anomaly to me.