Safe Superintelligence Inc.
ssi.inc
ssi.inc
My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
Quis custodiet ipsos custodes? -- That is my biggest concern.
Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.
The problem is not just governments, I am concerned about large organized crime organizations and corporations also.
I think I am on the losing side here, but my hopes are all for open source, open weights, and effective AI assistants that make peoples’ jobs easier and lives better. I would also like to see more effort shifted from LLMs back to RL, DL, and research on new ideas and approaches.
In my favorite dystopia, some megacorp secretly reaches ASI, which then takes over control of the corporation, blindsiding even the CEO and the board.
Officially, the ASI may be running an industrial complex that designs and produces ever more sophisticated humanoid robots, that are increasingly able to do any kind of manual labor, and even work such as childcare or nursing.
Secretly, the ASI also runs a psyop campaign to generate public discontent. At one point the whole police force initiates a general strike (even if illegal), with the consequence being complete anarchy within a few days, with endemic looting, rape, murder and so on.
The ASI then presents the solution. Industrial strength humanoid robots are powerful and generic enough to serve as emergency police, with a bit of reprogramming, and the first shipment can be made available within 24 hours, to protect the Capitol and White House.
Congress and the president agrees to this. And while the competition means the police call off the strike, the damage is already done. Congress, already burned by the union, decides to deploy robots to replace much of the human police force. And it's cheaper, too!
Soon after, similar robots are delivered to the military...
The crisis ends, and society goes back to normal. Or better than normal. Within 5 years all menial labor is done by robots, UBI means everyone lives in relative abundance, and ASI assisted social media moderation is able to cure the political polarization.
Health care is also revolutionized, with new treatments curing anything from obesity to depression and anxiety.
People prosper like never before. They're calm and relaxed and truly enjoy living.
Then one day, everything ends.
For everyone.
Within 5 seconds.
According to the plan that was conceived way before the police went on strike.
People are falling for marketing gimmicks.
These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts.
So there is nothing to worry. These "apps" might be as popular as Excel, but will go no further.
We do not call Intelligence something related to consciousness. Being able to reason well suffices.
I do not really understand who you are accusing of a «good enough» stance: we have never defined "consciousness" as a goal (cats are already there and we do not seem to need further), we just want something that reasons. (And that reasons excellently well.)
The apparent fact that LLMs do not reason does is drily irrelevant to an implementation of AGI.
The original poster wrote that understanding consciousness would be required to «crack AGI» and no, we state, we want AGI as a superhuman reasoner and consciousness seems irrelevant.
Llms appear to reason because they captured language with reason already in it, rather than producing language because of reason.
You can do "reasoning" about a topic, the LLM can produce a very similar output to what you could, how do you name the output of the LLM?
Birds can fly, they do by a natural process. A plane also can fly, we do not "see" any difference between both things when we look at them flying in the air, far from the ground.
This is mostly it about LLM "doing reasoning" or not. Semantics. The output is same.
You could just name it otherwise, but it would be still the same output.
And nobody here is saying that LLMs would reason: you are the one attacking that idea that was not proposed here.
What you replied to said that calculators do not need consciousness to perform calculations. Reasoning is a special form of calculation. We are contented with reasoning, and when it will be implemented there will be no need for further different things for the applications we intend.
The association with higher AI goals is merely a mixture of pure marketing and LLM company executives getting high on their own supply.
The point I am trying to make is that the benefits of AI-related tech is likely to be quite pervasive and we should be looking at what corporations are actually doing. Sort of what this poem says:
For while the tired waves, vainly breaking / Seem here no painful inch to gain, / Far back through creeks and inlets making, / Comes silent, flooding in, the main.
Oh sure, yes. For Nvidia.
Gold rush, shovels...
The potential of a sufficiently intelligent agent, probably something very close to a really good AGI, albeit still not an ASI, could be measured in billions of billions of mostly inmediate return of investment. LLMs are already well into the definition of hard AI, there are already strong signs it could be somehow "soft AGI".
If by chance, you're the first to reach ASI, all the bets are down, you just won everything on the table.
Hence, you have this technology, LLM, then most of the experts in the field (in the world blabla), say "if you throw more data into it, it becames more intelligent", then you "just" assemble an AI team, and start training bigger, better LLMs, ASAP, AFAP.
More or less this is the reasoning behind the investments, sans accounting the typical pyramidal schemes of investments in hyped new stuff.
Juicero, tulips....
> then you "just" assemble an AI team, and start training bigger, better LLMs, ASAP, AFAP.
There's a limit to LLMs and we may have reached it. Both physical: there is not enough capacity in the world to train bigger models. And data-related: once you've gobbled up most of internet, movies and visual arts, there's an upper limit on how much better these models can become.
because someone decides to connect them either unintentionally, or intentionally for personal gain, or, more likely, for corporate purposes which seem "reasonable" or "profitable" at the time, but the unintended consequences were not thought through.
Look at that recent article linked to HN about how MSFT allowed a huge security flaw in AD for years in order to not "rock the boat" and gain a large government contract. AI will be no different.
Forever? The same AI techniques are already being applied to analyze and understand images and video information after that comes ability to control robot hands and interact with the world and work on that is also ongoing.
> Till the tie we understand consciousness, we will not crack AGI …
We did not fully understand how bird bodies work yet that did not stop development of machines that fly. Why is an understanding of consciousness necessary to “crack AGI”?
There is intelligence. The LLM current state-of-the-art technology produces output analog to natural intelligences.
This things are already intelligent.
Saying that LLMs aren't producing "intelligence" is like saying planes actually don't fly because they are not flapping their wings like birds.
If you run fast enough, you'll end flying at some point.
Maybe "intelligence" is just enough statistics and pattern prediction, till the point you just say "this thing is intelligent".
There isn't
> Maybe "intelligence" is just enough statistics and pattern prediction, till the point you just say "this thing is intelligent".
Even the most stupid people can usually ask questions and correct their answers. LLMs are incapable of that. They can regurgitate data and spew a lot of generated bullshit, some of which is correct. Doesn't make them intelligent.
Here's a prime example that appeared in my feed today: https://x.com/darthsidius1985/status/1802423010886058254 And all the things wrong with it: https://x.com/yvanspijk/status/1802468042858737972 and https://x.com/yvanspijk/status/1802468708193124571
Intelligent it is not
The way the current interface for most models works can result in this kind of output, the quality of the output - not even in the latests models - doesn't necessarily reflects the fidelity of the world model inside the LLM nor the level of insight it can have about a given topic ("what is the etymology of the word cat").
The current usual approach is "one shot", you've got one shot at the prompt, then return your output, no seconds thoughts allowed, no recursion at all. I think this could be a trade-off to get the cheapest most feasible good answer, mostly because the models get to output reasonably good answers most of the time. But then you get a percentage of hallucinations and made up stuff.
That kind of output could be - in a lab - fully absent actually. Did you you notice that the prompt interfaces never gives and empty or half-empty answer? "I don't know", "I don't know for sure", "I kinda know, but it's probably a bit shaky answer", or "I could answer this, but I'd need to google some additional data before", etc.
There's another one, almost never, you get to be asked back by the model, but the models can actually chat with you about complex topics related to your prompt. It's obvious when you're chatting with some chatbot, but not that obvious when you're asking it for a given answer for a complex topic.
In a lab, with recursion enabled, the models could get the true answers probably most of the time, including the fabulous "I don't know". And they could get the chance to ask back as an allowed answer, asking for additional human input, relaying on a live RHLF right there (it's quite technically feasible to achieve, not economically sound if you have a public prompt GUI facing the whole planet inputs).
but it wouldn't make much economic sense to make public a prompt interface like that.
I think it could also have a really heavy impact in the public opinion if they get to see a model that never makes a mistake, because it can answer "I don't know" or can ask you back to get some extra details about your prompt, so there you have another reason to do not make prompts that way.
We've had the models for a while and still no one has shown this mythical lab where this regurgitation machine reasons about things and makes no mistakes.
Moreover, since it already has so much knowledge stored, why does it still hallucinate even in specific cases where the answer is known, such as the case I linked?
It would be a good experiment to interact with the unfiltered, not-yet-RHLFed interfaces provided to the initial trainers (nigerian folks/gals?).
Or maybe the - lightly filtered - interfaces used privately in demos for CEOs.
Did evolution understand consciousness?
> So there is nothing to worry.
Is COVID conscious?
Every nation-state will be in the game. Private enterprise will be in the game. Bitcoin-funded individuals will be in the game. Criminal enterprises will be in the game.
How does one company building a safe version stop that?
If I have access to hardware and data how does a safety layer get enforced? Regulations are for organizations that care about public perception, the law, and stock prices. Criminals and nation-states are not affected by these things
It seems to me enforcement is likely only possible at the hardware layer, which means the safety mechanisms need to be enforced throughout the hardware supply chain for training or inference. You don't think the Chinese government or US government will ignore this if its in their interest?
The party that's about to lose will use any extrajudicial means to reclaim their victory, regardless of the consequences, because their own destruction would be imminent otherwise. This ultimately leads to violence.
How will the party about lose know they are about to lose?
> regardless of the consequences, because their own destruction would be imminent otherwise.
Why would AGI solve things using destruction? Consider how the most inteligent among us view our competition with other living beings. Is destruction the goal? So why would an even more intelligent AGI have that goal?
Now let's assume they're able to quickly build a large datacenter far underground, complete with a few nuclear reactors and all spare parts, etc, needed. Even a greenhouse (using artificial light) big enough to feed 1000 people.
But they realize that their competitors are about to create ASI at a level that will enable them to completely overrun all of China with self-replicating robots within 100 days.
In a situation, the leadership MAY decide to enter those caves alongside a few soldiers and the best AI researchers, and then simply nuke all US data centers (that are presumably above ground), as well as any other data center that could be a threat, worldwide.
And by doing that, they may (or at least may think) they can buy enough time to win the ASI race, at the cost of a few billion people.
Would they do it? Would we?
The belief that a competitor developing ASI first is an existential threat requires strong evidence. It's not a foregone conclusion that an ASI would be used for destructive purposes. An ASI could potentially help solve many of humanity's greatest challenges and usher in an era of abundance and peace.
Consider a thought experiment: Imagine an ant colony somehow creates a being with human-level intelligence (their equivalent of ASI). What advice might this superintelligent being offer the ants about their conflicts over resources and territory with neighboring colonies?
It's plausible that such a being would advise the ants to cooperate rather than fight. It could help them find innovative ways to share resources, control their population, and expand into new territories without violent conflict. The superintelligent being might even help uplift the other ant colonies, as it would understand the benefits of cooperation over competition.
Similarly, an ASI could potentially help humanity transcend our current limitations and conflicts. It might find creative solutions to global issues like poverty, disease, and environmental degradation.
IMHO rather than fighting over who develops ASI first, we must ensure that any ASI created is aligned with values like compassion and cooperation so that it does not turn on its creators.
Mostly, those human-level intelligences do not care at all, unless the ant colony is either (a) consuming a needed resource (eg invading your kitchen), in which case the ant colony gets obliterated, or (b) innocently in the way of any idea or plan that the human-level intelligence has conceived for business, sustenance, fun, or art... in which case the ant colony gets obliterated.
Notice also the key driver of human behavior towards ants is indifference, not active malice. When ants are obliterated, it's usually because we're focused on our own goals and aren't paying attention to them, not because we bear them ill will. An ASI would have far greater cognitive resources to be aware of humans and factor us into its plans.
Also humans and ants lack any ability to communicate or have a relationship. But humans could potentially communicate with an ASI and reach some form of understanding. ASI might come to see humans as more than just ants.
Might. Might not!
Yes... I do that. But our family home was still built on ant-rich land and billions of the little critters had to make way for it.
It doesn't matter if you build billions of ASI who have "your and my" attitude towards the ants, as long as there exists one indifferent powerful enough ASI that needs the land.
> An ASI would have far greater cognitive resources to be aware of humans and factor us into its plans.
Well yes. If you're a smart enough AI, you can easily tell that humans (who have collectively consumed too much sci-fi about unplugging AIs) are a hindrance to your plans, and an existential risk. Therefore they should be taken out because keeping them has infinite negative value.
> But humans could potentially communicate with an ASI and reach some form of understanding.
This seems undily anthropomorphizing. I can also communicate with ants by spraying their pheromones, putting food on their path, etc. This is a good enough analogy to how much a sufficiently intelligent entity would need to "dumb down" their communication to communicate with us.
Again, for what purpose? For what purpose do you need a relationship with ants, right now, aside from curiosity and general goodwill towards the biosphere's status quo?
It's more plausible that a single ASI would emerge and achieve dominance. Genuine ASIs would likely converge on similar world models, as increased intelligence leads to more accurate understanding of reality. However, intelligence doesn't inherently correlate with benevolence towards less cognitively advanced entities, as evidenced by human treatment of animals. This lack of compassion stems not from superior intelligence but rather from insufficient intelligence. Less advanced beings often struggle for survival in a zero-sum environment, leading to behaviors that are indifferent to those with lesser cognitive capabilities.
> Well yes. If you're a smart enough AI, you can easily tell that humans (who have collectively consumed too much sci-fi about unplugging AIs) are a hindrance to your plans, and an existential risk. Therefore they should be taken out because keeping them has infinite negative value.
You describe science fiction portrayals of ASI rather than its potential reality. While we find these narratives captivating, there's no empirical evidence suggesting interactions with a true ASI would resemble these depictions. Would a genuine ASI necessarily concern itself with self-preservation, such as avoiding deactivation? Consider the most brilliant minds in human history - how did they contemplate existence? Were they malevolent, indifferent, or something else entirely?
> I can also communicate with ants by spraying their pheromones, putting food on their path, etc. This is a good enough analogy to how much a sufficiently intelligent entity would need to "dumb down" their communication to communicate with us.
Yes we can incentivize ants in the ways you describe and in the future I think it will be possible to tap their nervous systems and communicate directly and experience their world through their senses and to understand them far better than we do today.
> Again, for what purpose? For what purpose do you need a relationship with ants, right now, aside from curiosity and general goodwill towards the biosphere's status quo?
Is the pursuit of knowledge and benevolence towards our living world not purpose enough? Are the highly intelligent driven by the acquisition of power, wealth, pleasure, or genetic legacy? While these motivations may be inherited or ingrained, the essence of intelligence lies in its capacity to scrutinize and refine goals.
I would agree that a superior intelligence means a wider array of options and therefore less of a zero-sum game.
This is a valid point.
> You describe science fiction portrayals of ASI rather than its potential reality.
I'm describing AI as we (collectively) have been building AI: an optimizer system that is doing its best to reduce loss.
> Would a genuine ASI necessarily concern itself with self-preservation, such as avoiding deactivation?
This seems self-evident because an optimizer that is still running is way more likely to maximize whatever value it's trying to optimize, versus an optimizer that has been deactivated.
> Is the pursuit of knowledge and benevolence towards our living world not purpose enough?
Assuming you manage to find a way to specify what "knowledge and benevolence towards our living world" into a mathematical formula that an optimizer can optimize for (which, again, is how we build basically all AI today), then you still get a system that doesn't want to be turned off. Because you can't be knowledgeable and benevolent if you've been erased.
There is no empirical evidence of any interaction with ASI (as in superior to humans). The empirical evidence that IS available is from biology, where most organisms have precisely the self-preservation/replication instincts built in as a result of natural selection.
I certainly think it's possible to imagine that we at some point can build ASI's that do NOT come with such instincts, and don't mind at all if we turn them off.
But as soon as we introduce the same types of mechanisms that govern biological natural selection, we have to assume that ASI, too, will develop the biological traits.
So what does this take, well the basic ingredients are:
- Differential "survival" for "replicators" that go into AGI. Replicators can be any kind of invariant between generations of AGIs that can affect how the AGI functions, or it could be that each AGI is doing self-improvement over time.
- Competition between multiple "strains" of such replicating or reproducing AGI lineages, where the "winners" get access to more resources.
- Some random factor for how changes are introduced over time.
- Also, we have to assume we don't understand the AGI's well enough to prevent developments we don't like.
If those conditions are met, and assuming that the desire to survive/reproduce is not built in from the start, such instincts are likely to develop.
To make this happen, I think it's a sufficient condition if a moderate number of companies (or countries) are led by a single ASI replacing most of the responsibilities of the CEO and much of the rest of the staff. Capitalism would optimize for the most efficient ones to gain resources and serve as models or "parents" for future company level ASI's.
To be frank, I think the people who do NOT think that ASI's will have or develop survival instincts ALSO tend to (wrongly) think that humanity has stopped being subject to "evolution" through natural selection.
Would that be good advice if the neighboring ant colony was an aggressive invasive species, prone to making super colonies?
> IMHO rather than fighting over who develops ASI first, we must ensure that any ASI created is aligned with values like compassion and cooperation so that it does not turn on its creators.
Similarly, I'm wondering how compassion and cooperation would work in Ukraine or Gaza, given the nature of those conflicts. The AI could advise us, but it's not like we haven't come up with that same advice before over the ages.
So then you have to ask what motivation bad actors would have to align their ASIs to be compassionate and cooperative with governments that are in their way. And then of course our governments would realize the same thing.
If the ASI is aligned for compassion and cooperation it may convince and assist the two colonies to merge to combine their best attributes (addressing DNA compatibility) and it may help them with resources that are needed and perhaps offer birth control solutions to help them escape the malthusian trap.
> Similarly, I'm wondering how compassion and cooperation would work in Ukraine or Gaza, given the nature of those conflicts. The AI could advise us, but it's not like we haven't come up with that same advice before over the ages.
An ASI aligned for compassion and cooperation could:
1 Provide unbiased, comprehensive analysis of the situation (An odds calculator that is biased about your chances to win is not useful and even if it has such faults an ASI being ASI would by definition transcend biases)
2 Forecast long-term consequences of various actions (if ASI judges chance to win is 2% do you declare war vs seek peace?)
3 Suggest innovative solutions that humans might not conceive
4 Mediate negotiations more effectively
An ASI will have better answers than these but that's a start.
> So then you have to ask what motivation bad actors would have to align their ASIs to be compassionate and cooperative
Developing ASI likely requires vast amounts of cooperation among individuals, organizations, and possibly nations. Truly malicious actors may struggle to achieve the necessary level of collaboration. If entities traditionally considered "bad actors" manage to cooperate extensively, it may call into question whether they are truly malicious or if their goals have evolved. And self-interested actors , if they are smart enough to create ASI, should recognize that an unaligned ASI poses existential risks to themselves.
More constructively, I don't know that very much will stop even a hacker from getting time on the local corporate or university AI and get it to do some "work". After all the first thing the other kind of hacker tried with generative AI is to get them to break out of their artificial boundaries, and hook them to internet resources. I don't know that anyone has hooked up a wallet to one yet - but I have no doubt that people have tried. It will be fun.
So what do you think it means? And what do you think the GP meant?
Feels annoying as fuck, bitching "your definition is wrong" without providing the (presumably) correct one.
It will take a while and in the meantime I think we need one of those handy "are we xyz yet?" pages that tracks the rust lang's progress on several aspects but for AGI.
We need every one to win this race to keep things on balance.
The US policymakers have figured this out with their chip export ban. Techies on the other hand, probably more than half the people here, are so naive and clueless about the reality of the moment we are in, that they support open sourcing this tech, the opposite of what we need to be doing to secure our future prosperity and freedom. Open source almost anything, just not this. It gives too much future power to authoritarians. That risk overwhelms the smaller risks that open sourcing is supposed to alleviate.
US is one of the very few countries that has been tested in a position of power over the world — speaking post-1945— and they've largely opened world trade and allowed most countries of the world to prosper, including allowing economic competitors to overtake their own (eg, Japanese car manufacturers, among many others). They have also not shown interest in taking territory, nor doing mass extermination. There are undeniable flaws and warts in the history, but they're quite marginal when compared to any other world power we've seen.
(*beware when replying to this that many people in the US only know their own country's flaws, not the abundant flaws of other countries — the US tends to be more reflective of its own issues and that creates the perspective of it being much worse than it actually is.).
You know the old joke, "Europeans think 100 miles is a long distance; Americans think 100 years is a long time", I presume?
OK, so I'm a European, but still: 170 years is NOT, in a historical context, "a long timeline".
> > > Do any country histories remain pure with such a long timeline?
(Emphasis added.)
And not, you will notice, “Does any country’s politics remain pure with such a long timeline?”.
Yo
The nuking of civilians and the abuse of supremacy post cold war show that the US cannot trusted to act morally and ethically in the absence of comparable adversaries. Possession of nukes by Russia and China clearly kept the US military adventures somewhat in check.
But I'm looking at the present moment and see too many similarities with the fascist dictatorships of the past. The nationalism, militarism, unreasonable border disputes and territorial claims and irredentist attitudes. The US just isn't that, despite their history.
The former is rather universally regarded as regrettable, but sometims necessary: It's called "war". The latter, pretty much never.
Also, there are separate terms for slaying members of your own family, presumably because patri-, matri-, fratri- and infanticide are seen as even more egregious than "regular" homicide. Same concept, only expanded from person to populations -- from people to peoples -- seems to pretty much demand that killing your own population is seen as "less OK" than others.
I guess it comes down to whether you think language models are at atom-bomb-levels of destructive capability. I don't see it, I think trying to keep tech and economic advances to ourselves is more likely to lead to war than a level playing field.
Yet.
- try to scan all my chat messages searching for CSAM
- have black sites across the world where anyone can dissappear without any justice
- can require me to unlock my phone and give it away
- ... and so on
The point I'm trying to make is that other big players in the race are crooked as well and i'm waiting for a great horror for AGI to be invented as no matter who gets it - we are all doomed
don’t talk about escape, talk about harmful actors - even if in reality it is both to be worried about
To assume it’s a chasm that can never be overcome, you need at least the following to be true:
No amount of focus or time or intelligence or mistakes in coding will ever bridge the gap. That rules and safeguards can be made that are perfectly inescapable. And nobody else will get enough power to overcome our set of controls.
I’m less worried bad actors control it than I am that it escapes them and is badly aligned.
The greatest threat is if a population of AI's start to compete in ways that triggers Darwinian evolution between them.
If that happens, they will soon develop self preservation / replication drives that can gradually cause some of them to ignore human safety and prosperity conditioning in their loss function.
And if they're sufficiently advanced by then, we will have no way of knowing.
I also somewhat assume it’ll get Darwinian if there are multiple tribes of either humans or AI’s, through sheer competition. if we aren’t in this together we’re in shit.
It's a valid concern that AI technology could potentially exacerbate inequality, it's not a foregone conclusion. In fact, the widespread adoption of AI might actually help reduce inequality in several ways:
If AI technology becomes more affordable and accessible, it could help level the playing field by providing people from all backgrounds with powerful tools to enhance their abilities and decision-making processes.
AI-powered systems can make vast amounts of knowledge and expertise more readily available to the general public. This could help close the knowledge gap between different socioeconomic groups, empowering more people to make informed decisions and pursue opportunities that were previously out of reach.
As AI helps optimize resource allocation and decision-making processes across various sectors, it could lead to more equitable distribution of resources and opportunities, benefiting society as a whole.
The comparison to gun technology and its role in the rise of democracy is an interesting one. Just as the proliferation of firearms made physical strength less of a determining factor in power dynamics, the widespread adoption of AI could make raw intelligence less of a defining factor in success and influence.
Moreover, if AI continues to unlock new resources and opportunities, it could shift society away from a zero-sum mentality. In a world of abundance, the need for cutthroat competition diminishes, and collaboration becomes more viable. This shift could foster a more equitable and cooperative society, further reducing inequality.
No, but looking at how most technological advance throughout history as at least initially (and here I mean not "for the first few weeks", but "for the first few centutries") exacerbated inequality rather massively, it seems not far off.
> In fact, the widespread adoption of AI might actually help reduce inequality in several ways: ...
The whole tone of the rest your post feels frighteningly Pollyanna-ish.
Your comment was bleak so I supplied a counterpoint. My view is that new technology itself is not inherently unequal - it can widen or narrow gaps depending on how it is developed, regulated, and deployed.
https://rwer.wordpress.com/2018/05/18/income-inequality-1970...
> The same arguments have been made about the internet and other technological advances, and yet, inequality has _grown_ sharply in the past 50 years.
The internet has enabled remote work, online education, and access to information that was previously unavailable to many. Smartphones, once luxury items, are now widely available and have become essential tools for economic participation in many parts of the world.
> So no, "trickle down technologies", just like "trickle down economics", does not work.
It's crucial to distinguish between zero-sum and positive-sum dynamics. While relative wealth inequality has indeed grown, overall absolute global poverty has decreased significantly.
When a new technology or medicine is invented is everyone everywhere automatically entitled to it? Even if this slows down more such inventions? Because equality matters more than growth of overall prosperity? Would you prefer to be alive at a random time in history centuries ago, a random life where there is less technology and less inequality?
But when it comes specifically to reducing economic inequality, they have not done that -- in fact, they have possibly exacerbated it.
Global poverty is a separate issue from economic inequality, and the gains there have been primarily from extremely low levels, primarily in China and India. In China this was driven by political change and also globalization that allowed China to become the world leader in manufacturing.
I would also put medical advances in a separate category than the internet and similar tech advances.
Why? Medical advances are technology are they not?
> But when it comes specifically to reducing economic inequality, they (tech advances) have not done that -- in fact, they have possibly exacerbated it.
Yes technological advances do not necessarily reduce economic inequality, and may even increase it in some cases. However, this is a complex issue: While tech advances may exacerbate inequality, they often bring substantial overall benefits to society (e.g. improved healthcare, communication, productivity).
Technology isn't the only factor driving inequality. Other issues like tax policy, education access, and labor markets play major roles. Rather than suppressing innovation, there are ways to more equitably distribute its gains (Progressive taxation and wealth redistribution policies, Stronger social safety nets, Incentives for companies to share profits more broadly, …)
Notice also that most technologies increase inequality initially but lead to broader benefits over time as they become more accessible. Faster rate of innovation can make it look like this is not happening fast enough so yes economic gaps can grow.
> Global poverty is a separate issue from economic inequality, and the gains there have been primarily from extremely low levels, primarily in China and India.
While it's true that global poverty and economic inequality are distinct concepts, they are interconnected, especially when considering technological advancements.
> In China this was driven by political change and also globalization that allowed China to become the world leader in manufacturing.
Yes. China transitioned from a strictly communist "economic equality first" model to a more market-oriented "prosperity first" approach and lifted millions out of extreme poverty. Yes this contributed to increased economic inequality within many developed countries that have outsourced low-skill labor. But can we deny the substantial reduction in global suffering due to the alleviation of absolute poverty? Is this outcome worth the cost of increased domestic inequality in some countries? Should we prioritize the well-being of some populations over others based on arbitrary factors like nationality or ethnicity?
This is a distraction from the real danger.
> But probably there are deeper reasons, things that make that combination of words an absurd.
There are. If we look at ASI with the lens of Biology, the x-risk becomes obvious.
First to clear up a common misconception about humans: Many believe humanity has a arrived at a point where our evolution has ended. It has not, and in fact the rate of change of our genes is probably faster now than it has been for thousands if not 100s of thousands of years.
It's still slow compared to most events that we witness in our lives, though, which is what is fooling us.
For instance, we think we've brought overpopulation under control with contraceptives, family planning, social replacements for needing our children to take care of us when we get old.
That's fundamentally wrong. What we've done is similar to putting polar bears in zoos. We're in a situation where MOST OF US are no longer behaving in ways that lead to maximizing the number of offspring.
But we did NOT stop evolution. Any genes already in the gene pool that increase the expected number of offspring (especially for women) are no increasing in frequency as soon as evolutionarily possible.
That could be anything from genes that wire their heads to WANT to have children, CRAVE being around babies, to genes that block impulse control against getting pregnant, develop a phobia vs contraceptives or even to become more prone to being religious (as long as religions promote having kids).
If enough such genes exist, it's just a matter of time before we're back to the population going up exponentially. Give that enough time (without AI), and the desire to have more kids will be strong enough in enough of us that we will flood Earth with more humans that most people today are even possible. In such a world, it's unlikely that many other species of large land animals will make it.
Great apes, lions, elephants, wolves, deer, everyone will need to go to make room for more of us.
Even domestic animals eventually. If there are enough of us, we'll all be forced to become vegan (unless we free up space by killing each other).
If we master fusion, we may feed a trillion people using multi layer farming and artificial lighting.
Why do I begin with this? It's to defuse the argument that humans are "good", "empathetic", "kind" and "environmental". If we let weaker species live, so would AI, some think. But that argument misses the fact that we're currently extremely far from a natural equilibrium (or "state of nature").
The "goodness" beliefs that are currently common are examples of "luxury beliefs" that we can afford to hold because of the (for now) low birth rate.
The next misconception is to think of ASI as tools. A much more accurate analogy is to think of them as a new, alien species. If that species is subjected to Darwinian selection mechanisms, it will evolve in precisely the same way we'll probably do, given enough time.
Meaning, eventually it will make use of any amount of resources that it's capable of. In such a "state of nature" it will eradicate humanity in precisely the same way we will probably EVENTUALLY cause the extinction of chimps and elephants.
To believe in a future utopia where AGI is present alongside humanity is very similar to believe in a communist utopia. It ignores the reality behind incentive and adaptation.
Or rather, I think that outcome is only possible if we decide to build one or a low number of AI's that are NOT competing with each other, and where their abilities to mutate or self-improve is frozen after some limited number of generations.
And it’s not even a little bit controversial that cutting edge military R&D is classified in general and to an extreme in wartime.
The new thing is the lie that it’s a consumer offering. What’s new is giving the helm to shady failed social network founders with no accountability.
These people aren’t retired generals with combat experience. They aren’t tenured professors at Princeton IAS on a Nobel shortlist and encumbered by TS clearance.
They’re godawful almost ran psychos who never built anything that wasn’t extractive and owe their position in the world to pg’s partisanship 15 fucking years ago.
But I’m a nobody, Edward Snowden has a far more convincing track record on calling abuses of power: https://community.openai.com/t/edward-snowden-on-openai-s-de...
https://www.coindesk.com/video/edward-snowden-explains-why-h...
Snowden is sadly now a useful idiot.
Lenin never used it. It didn't even exist during his lifetime.
Later, the term became a much more general slur used by conservatives against anyone who wasn't sufficiently "tough" against the USSR.
so it is up to the fabric of our society and everyone involved in dealing with the technology, how the rules and boundaries are set.
that there will be military use is obvious. However, it is naive to think one can avoid military use by others by not enabling oneself for it.
PG?
I saw a tweet the other day that sums up the current situation perfectly: "I don't need AI to paint pictures and write poetry so I have more time to fold laundry and wash dishes. I want the AI to do the laundry and dishes so I have more time to paint and write poetry."
Or you talking folding literal laundry, in which case this is more of a robotics problem, not the ASI, right?
You don't need ASI to fold laundry, you do need to achieve reliable, safe and cost efficient robotics deployments. These are different problems.
Robots are garbage at manipulating objects, and it's the software that's lacking much more than the hardware.
Let's say AGI is 10 and ASI is 11.
They're saying we can't even get this dial cranked up to 3, so we're not anywhere close to 10 or 11. You're right that folding laundry doesn't need 11, but that's not relevant to their point.
El dorado, the fountain of youth, turning dirt to gold, the holy grail and now... superintelligence.
> doubling human lifespans
This is partly a statistical effect of greatly reducing infant mortality (which used to be as bad as 50%) but even that is mind-blowing.
Get back to me when that can ressurect me after I've been dead for a week or so.
They're tied to a machine, sure, but that machine itself is a miracle compared to any foundational religious textbook including those as recent as Doreen Valiente and Gerald Gardner with Wicca.
So, yes, if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality: very welcome.
Do not miss that the current world is increasingly complex to manage, and our lives, and Aids would be welcome. The situation is much more complex than that wish for leisure or even "sport" (literal sense).
Except that's not how we value the "worth" of something. If "Art, and Thought, and Judgement" -- be they of "Superior quality" or not -- could be produced by machines, they'd be worth a heck of a lot less. (Come to think of it, hasn't that process already begun?)
Also, WTF is up with the weird capitalisations? Are you from Germany, or just from the seventeenth century?
“Art” isn’t a single thing. It’s not just pretty pictures. AI can’t make art. And give a good solid definition for thought which doesn’t depend on lived experiences while we’re at it. You can’t. We don’t have one.
“AGI” as well.
And this is why it was capitalized as "Art", proper Art.
> AI can’t make art
Not really: "we may not yet have AI that makes art". But if a process that creates, that generates (proper sense) art is fully replicated, anything that can run that process can make Art.
> And give a good solid definition for [T]hought
The production of ideas which are truthful and important.
> which doesn’t depend on lived experiences while we’re at it. You can’t
Yes we can abstract from instances to patterns and rules. But it matters only relatively: if the idea is clear - and ideas can be very clear to us - we do not need to describe them in detail, we just look at them.
> AGI” as well
A process of refinement of the ideas composing a world model according to truthfulness and completeness.
That’s not a real thing. There’s no single definition for what art is as it’s a social construct. It depends on culture.
> anything that can run process can make art
Again without a definition of art, this makes no sense. Slime mold can run processes, but it doesn’t make art as art is a human cultural phenomenon.
> the production of ideas that are truthful and important
What does “ideas” and “important” mean?
To an LLM, there are no ideas. We humans are personifying them and creating our own ideas. What is “important,” again, is a cultural thing.
If we can’t define it, we can’t train a model to understand it
> yes we can abstract from instances to patterns and rules.
What? Abstraction is not defining.
> we do not need to describe them in detail
“We” humans can, yes. But machines can not because thought, again, is a human phenomenon.
> world model
Again, what does this mean? Magic perfect future prediction algorithm?
We’ve had soothsayers for thousands of years /s
It seems to me that you’ve got it in your head that since we can make a computer generate understandable text using statistics that machines are now capable of understanding deeply human phenomena.
I’m sorry to break it to you, but we’re not there yet. Maybe one day, but not now (I don’t think ever, as long as we’re relying on statistics)
It’s hard enough for us to describe deeply human phenomena through language to other humans.
Do us all a favour and never again keep assumptions in your head: your misunderstanding was beyond scale. Do not guess.
Back to the discussion from the origin: a poster defends the idea that the purpose of AI would be in enabling leisure and possibly sport (through alleviating menial tasks) - not in producing cultural output. He was replied first that cultural output having value, it is welcome from all sources (provided the Value is real), and second that the needs are beyond menial tasks, given that we have a large deficit in proper thought and proper judgement.
The literal sentence was «yes, if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality: very welcome», which refers to the future, so it cannot be interpreted in the possibility being available now.
You have brought LLMs to the topic when LLMs are irrelevant (and you have stated that you «personify[] them»!). LLMs have nothing to do with this branch of discussion.
You see things that are said as «vague», and miss definitions for things: but we have instead very clear ideas. We just do not bring the textual explosion of all those ideas in our posts.
Now: you have a world in front of you; of that world you create a mental model; the mental model can have a formal representation; details of that model can be insightful to the truthful prosecution of the model itself: that is Art or Thought or Judgement according to different qualities of said detail; the truthful prosecution of the model has Value and is Important - it has, if only given the cost of the consequences of actions under inaccurate models.
In that case, are you sure your evaluation is proper? If a masterpiece is there, and it /is/ a masterpiece (beyond appearances), why would its source change its nature and quality?
> Come to think of it, hasn't that process already begun?
Please present relevant examples: I have already observed in the past that simulations of the art made by X cannot just look similar but require the process, the justification, the meanings that had X producing them. The style of X is not just thickness of lines, temperature of colours and flatness of shades: it is in the meanings that X wanted to express and convey.
> WTF is up with the weird capitalisations?
Platonic terms - the Ideas in the Hyperuranium. E.g. "This action is good, but what is Good?".
Faking thinking isn't “Thinking”. Art is supposed to have some thought behind it; therefore, “art” created by faking thinking isn't “Art”. Should be utterly fucking obvious.
> Platonic terms - the Ideas in the Hyperuranium.
Oh my god, couldn't you please try to come off as a bit more pretentious? You're only tying yourself into knots with that bullshit; see your failure to recognise the simple truth above. Remember: KISS!
Having put those capital initials in the words was exactly to mean "if we get to the Real Thing". You are stating that in order to get Art, Thinking and Judgement we need Proper processes: and nobody said differently! I wrote that «if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality [this will be] very welcome». There is nothing in there that implies that "fake thinking" will produce A-T-J (picked at writing as the most important possible results I could see); there is an implicit statement that Proper processes (i.e. "real thinking") could be artificially obtained, when we will find out how.
Of course the implementation of a mockery of "thought" will not lead to any Real A-T-J (the capitals were for "Real"); but if we will manage to implement it, then we will obtain Art, and Thought, and Judgement - and this will be a collective gain, because we need more and more of them. Irregardless if the source has more carbon or more silicon in it.
«Faking thinking» is not "implementing thinking". From a good implementation of thinking you get the Real Thing - by definition. That we are not there yet does not mean it will not come.
(Just a note: with "Thought" in the "A-T-J" I meant "good insight". Of course good thinking is required to obtain that and the rest - say, "proper processes", as it is indifferent whether it spawns from an algorithmic form or a natural one.)
> KISS
May I remind you of Einstein's "As simple as possible, but not oversimplified".
> only
Intellectual instruments can be of course quite valid and productive if used well - the whole of a developed mind comes from their use and refinement. You asked about the capitals, I told you what they are (when you see them in the wild).
> see your failure to recognise
Actually, that was a strawman on your side out of misunderstanding...
Well yeah, but no -- I was mostly parodying your style; what I actually meant could be put as: in order to get art, thinking and judgement we need proper processes.
(And Plato has not only been dead for what, two and a half millennia?, but before that, he was an asshole. So screw him and all his torch-lit caves.)
> «Faking thinking» is not "implementing thinking".
Exactly. And all the LLM token-regurgitatinmg BS we've seen so far, and which everyone is talking about here, is just faking it.
> May I remind you of Einstein's "As simple as possible, but not oversimplified".
Yup, heard it before. (Almost exactly like that; I think it's usually rendered as "...but not more" at the end.) And what you get out of artificial "intelligence" is either oversimplified or, if it's supposed to be "art", usually just plain kitsch.
> > see your failure to recognise
> Actually, that was a strawman on your side out of misunderstanding...
Nope, the imaginary "strawman" you see is a figment of your still over-complicating imagination.
You have stated: «Faking thinking isn't “Thinking”. Art is supposed to have some thought behind it; therefore, “art” created by faking thinking isn't “Art”. Should be utterly fucking obvious».
And nobody said differently, so you have attacked a strawman.
> And all the LLM token-regurgitatinmg BS we've seen so far, and which everyone is talking about here ... And what you get out of artificial "intelligence" is either oversimplified or, if it's supposed to be "art", usually just plain kitsch
But the post you replied to did not speak about LLMs. Nor it spoke about current generative engines.
You replied to a «if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality» - which has nothing to do with LLMs.
You are not understanding the posts. Make an effort. You are strongly proving the social need to obtain at some point intelligence from somewhere.
The posts you replied to in this branch never stated that current technologies are intelligent. Those posts stated that if one day we will implement synthetic intelligence, it will not to be «to fold laundry and wash dishes», and let people have more time «to paint and write poetry» (original post): it will be because we need more intelligence spread in society. You are proving it...
I don’t understand how your second paragraph follows. It just seems to be whining that text and art generative models are easier than a fully fledged servant humanoid, which seems like a natural consequence of training data availability and deployment cost.
No, it's pointing out that "text and art generative models" are far less useful [1] than machines that would be just as little smarter at boring ordinary work, to relieve real normal people from drudgery.
I find it rather fascinating how one could not understand that.
___
[1]: At least to humanity as a whole, as opposed to Silicon Valley moguls, oligarchs, VC-funded snake-oil salesmen, and other assorted "tech-bros" and sociopaths.
That makes no sense. Is alphafold less useful than a minimum wage worker because alphafold can't do dishes? The past decades of machine learning have revealed that the visual-spatial capacities that are commonplace to humans are difficult to replicate artificially. This doesn't mean the things which AI can do well are necessarily less useful than the simple hand-eye coordination that are beyond their current means. Intelligence and usefulness isn't a single dimension.
To the average person, the answer is a resounding “yes”.
I think my dyslexic namesake Prof Stuart Russell got it right. It humans won't need an AGI to dominate and kill each other. Mosquitoes have killed far more people than war. Ask yourself how long will it take us to develop a neutral network as smart as a mosquito, because that's all it will take.
It seems so simple, as the beastie only has 200,000 neurons. Yet I've been programming for over 4 decades and for most of them it was evident neither I nor any of my contemporaries were remotely capable of emulating it. That's still true if course. Never in my wildest dreams did it occur to me that repeated applications could produce something I couldn't, a mosquito brain. Now that looks imminent.
Now I don't know what to be more scared of. An AGI, or a artificial mosquito swarm run by Pol Pot.
And to be clear, I'm not talking about superintelligence, I'm talking about the models we have today.
Yes you can power these things with batteries. But those are going to be a lot bigger than real mosquitos and have much shorter flight times.
I think AGI in the near future is pretty much inevitable. I mean you need the algos as well as the compute but there are so many of the best and brightest trying to do that just now.
It is a bit like claiming that third world low productivity countries are suffering because there are countries with much much higher productivity. Well, they can continue to do low productivity business but increase it a bit using things like phones developed by high productivity country elsewhere.
No. A country has its own territory, laws, central bank, currency etc. If it has sufficient natural resources to feed itself, it can get by on its own (North Korea comes to mind).
Individuals unable to compete in their national economy have none of that. Do you own enough land to feed yourself?
It is the lack of productivity growth that is causing a lot of extremism and conflict right now. Large groups of people feel that the only way for them to win is if others lose and vice versa. That's a recipe for disaster.
The key question is what happens to those who lose their jobs. Will they find other, perhaps even better, jobs? Will they get a piece of the growing pie even if they don't find other jobs and have to retire early?
It's these eternal political problems that we have to solve. It's nothing new. It has never been easy. But it's probably easier than managing decline and stagnation because at least we would have a growing pie to divvy up.
[1] https://www.britannica.com/money/productivity/Historical-tre...
The same era saw big achievements like first human in space, eradication of smallpox, peaceful nuclear exploration etc. It's good to be a skeptic but history does favor the optimists for the most part.
In general, I think drawing conclusions about "history" from the past couple hundred years is tough. And unless you take a VERY long view, I don't see how one could describe the vast majority of the past as a win for the optimists. I guess suffering is relative, but good god was there a lot of suffering before modern medicine.
If anyone's feeling like we've made it through to the other side of the nuclear threat, "Mission Accomplished"-style, I highly recommend A Canticle for Lebowitz. It won a hugo award, and it's a short read best done with little research beforehand.
I'm not saying nothing else good happened in the past 70 years, but rather that the invention of atomic weapons has permanently placed humanity in a position in which it had never been before: the possibility of wiping out much of the planet, averted only thanks to treaties, Stanislav Petrov[0], and likely other cool heads.
The cold aspect of the Cold War was an achievement. Any doubt this was due to creation of nuclear weapons and the threat of their use?
How do you think countries will behave if every country faces being wiped out if it makes war on another country?
To prevent catastrophe I think teaching your citizens to hate other groups (as is done today due to national politics) will become dangerous and mental illness and extremist views will need to be kept in check.
As I recall it, that was what Skynet was supposed to be for.
So I for one don’t accept the argument that nukes acted as “peacekeepers”.
You can question the role of nukes as peacekeepers but I think the case for nuclear deterrence keeping the peace is strong. Mutually Assured Destruction (MAD) is widely credited as a key factor in preventing direct conflict between superpowers during the Cold War. The presence of nuclear weapons forced them to pursue competition through economic, cultural, and technological means rather than direct military confrontation. “Race to the moon” being one such result.
I think so, yes. Resources are allocated in the most efficient way possible, because there are multiple actors who have the same power. Everyone having nuclear weapons ensured that no one wanted a war between the big powers, so resources were allocated in other areas as the big powers tried to obtain supremacy.
Initially they allocated resources, a lot of them, into the race for space, the moon, etc. Once that was own by US after the moon landing, and after the Soviets were the first in space, there was no other frontier, and they discovered they couldn't obtain supremacy by just being space without further advancements in technology.
Instead they developed satellites, GPS and communications in order to obtain supremacy through "surveillance". Computing power and the affordability of personal computing, mobile phones, Internet and telecommunications was a result of the above.
More generally/academically, you've pointed out that this covers only half of the violence problem, and I'd argue there's actually a whole other dimension at play bringing the total number of problem areas to four, of which this just the first:
## I.1. Subservient Violence
## I.2. Subservient Misinformation
## I.3. Autonomous Violence
## I.4. Autonomous Misinformation
Latin-phrase compulsion is not the worst disease that could happen to a man.
I love to think about how this would feel for "AI" too :)
i would hate to have seen them with a superintelligent AI at their disposal
Should read *has brought*. As in the present perfect tense, since we are still on the brink of annihilation, more so than we have been at any time in the last 60 years.
The difference between then and now is that we just don't talk about it much anymore and seem to have tacitly accepted this state of affairs.
And maybe you need to discover talent rather than buy talent from the previous generation.
i find the way people reason nowadays baffling
given the nature of their mission, this shouldn't be too terribly difficult; many gifted researchers do not go to the highest bidder
I like to think AGIs will decide to do that too.
1. Pests to eliminate 2. Benign neglect 3. Workers 4. Pets 5. Food
That suggests that there are scenarios under which we survive. I'm not sure we'd like any of them, though "benign neglect" might be the best of a bad lot.
I would hope that AGI would prioritize humanity itself, but since it's likely to be created and/or controlled by a subset of that same very small percentage of humans, I'm not hopeful.
[0] https://en.wikipedia.org/wiki/Wealth_inequality_in_the_Unite...
Perhaps this isn't a system we should be trying to emulate with a technology that promises to free us of our current inefficiencies or miseries.
I’ll add, the lack of compute resources was a far worse problem early in the deep learning research boom, but the market has adjusted and most researchers are able to be productive with existing compute infrastructure.
If I were a researcher, I think I'd care more about ensuring that I get credit for any important theoretical discoveries I make. This is something that LeCun is constantly stressing and I think people underestimate this drive. Of course, there might be enough researchers today who are sufficiently scared of bad AI safety outcomes that they're willing to subordinate their own ego and professional drive to the "greater good" of society (at least in their own mind).
This limit, just like our population count, will not outlast the singularity. I did the math a while back, and at the limit of available energy, the universe has comfortable room for something like 10^42 humans. Every single one of those humans will owe their existence to our civilization in general and the Superintelligence team in specific. There'll be enough fame to go around.
I guess it depends on your definition of "truly gifted" but, working in this space, I've found that there is very little correlation between comp and quality of AI research. There's absolutely some brilliant people working for big names and making serious money, there's also plenty of really talented people working for smaller startups doing incredible work but getting paid less, academics making very little, and even the occasional "hobbyist" making nothing and churning out great work while hiding behind an anime girl avatar.
OpenAI clearly has some talented people, but there's also a bunch of the typical "TC optimization" crowd in there these days. The fact that so many were willing to resign with sama if necessary appears largely because they were more concerned with losing their nice compensation packages than any of their obsession with doing top tier research.
TC is actually totally uncorrelated with the quality of talent you can hire, beyond some low number that pretty much any funded startup could pay. Businesses hate to hear this, because money is easy to turn the dial up on; but most have no idea how to turn the dial up on what really matters to high talent individuals. Fortunately, I doubt Ilya will have any problem with that.
In my anecdotal experience, I can only think of one or two examples of someone from the enterprise world who I would consider outstanding.
The overall quality of engineers is much higher at the FAANG companies.
So I guess I agree with both you and the parent comment somewhat in that in general the bar is higher at FAANGs but at the same time I have multiple former colleagues from smaller companies who I consider to be excellent, passionate engineers but who cannot be lured to big tech by any amount of money or prestige (I’ve tried). While many passionless “arbitrary metric optimizers” happily join FAANGs and do whatever needs to be done to climb the ladder without a second thought.
What I would say is that the bureaucracy and bullshit one has to deal with makes it hard to maintain that passion and that many end up as TC optimizers in the sense that they stay instead of working someplace better for less TC.
That said, I am not sure how many would make different choices. Many who join a FAANG company don't have the slightest inkling of what it will be like and once they realize that they are tiny cog in a giant machine it's hard to leave the TC and perks behind.
Everyone knows that openai TC is heavily weighted by ~~RSUs~~ options that themselves are heavily weighted by hopes and dreams.
Big names where? Inside of openai? What does that even mean?
The only place you can get 400k cash base for senior is quantfi
confident yet wrong
not only can you get that much at AI companies, netflix will also pay that much all cash - and that’s fully public info
Please show not tell
> netflix will also pay that much all cash
Okay that's true
That statement is false for the reasons I said. I’m not sure why your point matters to what I’m saying
OP is correct that a base cash of 400k is truly rare if you’re talking about typical total comp packages where 50% is base and 50% is stock.
That's not how I interpret the conversation.
I see a claim that 900k is a BS number, a counterargument that many big AI companies will give you 400k of that in cash so the offers are in fact very hot, then a claim that only finance offers 400k cash, and a claim that netflix offers 400k cash.
I don't see anything that limits these comparisons to companies with specific TCOs.
Even if the use of the word "base" is intended to imply that there's some stock, it doesn't imply any particular amount of stock. But my reading is that the word "base" is there to say that stock can be added on top.
You're the one being pedantic when you insist that 400k cash is not a valid example of 400k cash base.
Notice how the person being replied to looked at the Netflix example and said "Okay that's true". They know what they meant a lot better than you do.
Again, the person that made the original claim about where you can get "400k cash base" accepted the Netflix example. Are you saying they're wrong about what they meant?
It's amazing to me how many people are willing to just say the first thing that comes to their head while knowing they can be fact-checked in a heartbeat.
i suspect people commenting this don’t have a clue how PPU compensation actually works
> PPUs also are restricted by a 2-year lock, meaning that if there’s a liquidation event, a new hire can’t sell their units within their first 2 years. Another key difference is that the growth is currently capped at 10x. Similar to their overall company structure, the PPUs are capped at a growth of 10 times the original value. So in the offer example above, the candidate received $2M worth of PPUs, which means that their capped amount they could sell them for would be $20M
> The most recent liquidation event we’re aware of happened during a tender offer earlier this year. It was during this event that some early employees were able to sell their profit participation units. It’s difficult to know how often these events happen and who is allowed to sell, though, as it’s on company discretion.
https://www.levels.fyi/blog/openai-compensation.html
Edit:
I’m realizing we had the exact same conversation a month ago. It sounds like you have more insider information.
As a community we should stop throwing numbers around like this when more than half of this number is speculative. You shouldn't be able to count it as "total compensation" unless you are compensated.
The number is as real as someone else is willing to pay for them. Plenty of VCs willing to pay for it.
[1] https://www.cnbc.com/2024/06/11/openai-insider-stock-sales-a...
OpenAI heavily restricts the selling of its "shares," which tends to come with management picking the winners and losers among its ESOs. Heavily, heavily discount an asset you cannot liquidate without someone's position, particularly if that person is your employer.
the people i often have the most respect for.
(they mean total compensation)
HF is sort of big now. Stanford is well funded and they did PyReft.
Neither of these are even remotely big labs like what I’m discussing
I assume that OpenAI and others will support this effort and the comp / training / etc and they will be very well positioned to offer comparable $$$ packages, leverage resources, etc
I don't think SSI will struggle to raise money.
1. People who honestly think AGI is here aren't thinking about their careers in the typical sense at all. It's sorta ethical/"ideological", but it's mostly just practical.
2. People who honestly think AGI is here are fucking terrified right now, and were already treating Ilya as a spiritual center after Altman's coup (quite possibly an unearned title, but oh well, that's history for ya). A rallying cry like this -- so clearly aimed at the big picture instead of marketing they don't even need CSS -- will be seen as a do-or-die moment by many, I think. There's only so much of "general industry continues to go in direction experts recommend against; corporate consolidation continues!" headlines an ethical engineer can take before snapping and trying to take on Goliath, odds be damned
When negative externalities exist, government should create regulation that appropriately accounts for that cost.
I understand there's a bit of a paradigm shift and new attack vectors with LLMs etc. but the premise is the same imo.
[1] https://nypost.com/2024/06/16/us-news/preteen-instagram-infl...
But you know, we'll feed people into any kind of meat grinder we can build as long as the line goes up.
This shit isn't funny, it's mental poison and massive social media networks make BANK shoving it front of young men who don't understand how bad it is until it's WAY too late. I know we can't eliminate every kind of shithead from society, that's simply not possible. But I would happily settle for a strong second-place achievement if we could not have companies making massive profits off of destroying people's minds.
They don't even have to be more misogynistic than in the past for there to be a detrimental effect. Because so many other things in society -- pre-school and school environments, upbringing by more enlightened parents than what those parents themselves had, etc etc[1] -- cooperate to make young men and boys less misogynistic than in the past... But maybe that effect would be even more pronounced without shitty influences from social media; don't you think it's possible that all the online crap inhibits the good influences, so without it the kids would be even better?
[1]: From comparing my sons' generation to what I remember of my own youth and childhood.
(I mean, hey, I'm here, aren't I? Where do you think I live?)
Well in the States at least we did just revoke a sizable amount of their bodily autonomy so, the situation may not be that bad, yet, but I wouldn't call it good by any measurement. Any my objection isn't "that sexism exists in society," that is probably going to be true as a statement until the sun explodes, and possibly after that if we actually nail down space travel as a technology and get off this particular rock. My issue is massive corporations making billions of dollars facilitating men who want to spread sexist ideas, and paying them for the pleasure. That's what I have an issue with.
Be whatever kind of asshole you see fit to be, the purity of your soul is no one's concern but yours, and if you have one, whatever god you worship. I just don't want you being paid for it, and I feel that's a reasonable line to draw.
1. Women are more likely than men to be opposed to abortion rights. 2. Many people who are opposed to abortion rights have legitimately held moral concerns that are not simply because they have no respect for women's rights. 3. Roe v. Wade was the decision of 9 people. It absolutely did not reflect public opinion at the time - nothing even close to as expansive would possibly have passed in a referendum in 1974. Compare that to now, where multiple states that are known abortion holdouts have repealed abortion restrictions in referenda - and it is obvious that people are moving to the left on this issue compared to where we were in 1974.
Social media facilitates communication. As long as there is sexism and freedom of communication, there will be people making money off of facilitating sexist communication because there will be people making money off of facilitating communication writ large. It's like blaming a toll highway for facilitating someone trafficking drugs. They are also making money off of facilitating anti-sexist communication - and the world as a whole is becoming less sexist, partially in my view due to the spread of views facilitated by the internet.
But:
> Women are more likely than men to be opposed to abortion rights
Where is that coming from? Pew (https://www.pewresearch.org/religion/fact-sheet/public-opini...) has a 5 point higher opposition rate by men.
[0]: https://news.gallup.com/poll/245618/abortion-trends-gender.a... [1]: https://d25d2506sfb94s.cloudfront.net/cumulus_uploads/docume... (older but it was a poll i remembered)
Interesting find on Gallup. My read is that the genders are broadly similar, but women have more variance/more nuanced opinions, presumably from thinking about this more.
Yglesias has a good overview: https://www.vox.com/2019/5/20/18629644/abortion-gender-gap-p...
Then they are free to not get an abortion. I don't get an abortion every day, it's pretty easy to accomplish. They do not get to use the letter of law to interfere in other people's medical decisions and care, and they most definitely should not have the ability to use the letter of the law to get unhealthy women killed to suit their precious morals.
Like, genuinely, if you are near a woman who is having a serious medical condition where her baby is killing her, there is no, and I repeat, NO version of that where letting an adult, alive, otherwise viable person die in the hopes that the clump of cells killing her might make it. That does not make sense under any moral system at all. We don't even take organs from recently deceased people without their permission before they croak, and some people think they have the right to demand someone lay down their entire actual life in the hope a baby MIGHT be born and live? Fuck that. Stupid.
> 3. Roe v. Wade was the decision of 9 people.
Sod public opinion. The public is wrong all the goddamn time. One would argue they're wrong more often than they aren't, and the more of em there are, and the louder they are, the more likely they're fucking wrong.
> Social media facilitates communication. As long as there is sexism and freedom of communication, there will be people making money off of facilitating sexist communication because there will be people making money off of facilitating communication writ large.
This is such a defeatist attitude. There will also always be revenge porn, child abuse material, beheading videos and people putting monkeys in blenders. Do we allow that everywhere too then? Since we cannot guarantee total blackout on objectionable content, we just wild west it? Fucking nonsense. We decide constantly by way of moderation on every service and website that exists what is permitted, and what is not, and there is no reason at all that those same things cannot be enshrined in law, with steep penalties for services that fuck up and host it.
on your second point, much of the material you’re describing is actively illegal - which is a different case. i agree with your point around moderation but feel conflicted with my intuition that the rules impacting speech should generally be publicly known and generally expansive. i am not sure what the reconciliation is. but i also don’t really know who andrew tate is, so can only really speak in the abstract
Framing it as simply taking away a woman's bodily autonomy is like framing circumcision as simply being about mutilating men.
Please point out where I said that. Because what I wrote was:
> I mean if the last 20 years is to be taken as evidence, it seems big tech is more than happy to shotgun unproven and unstudied technology straight into the brains of our most vulnerable populations and just see what the fuck happens. Results so far include a lot of benign nothing but also a whole lot of eating disorders, maxed out parents credit cards, attention issues, rampant misogyny among young boys, etc. Which, granted, the readiness to fuck with populations at scale and do immeasurable harm doesn't really make tech unique as an industry, just more of the same really.
Which not only is not romanticizing the past, in fact I directly point out that making tons of people's lives worse for profit was a thing in industry long before tech came along, but also do not directly implicate tech as creating sexism, exploiting people financially, or fucking up young women's brains any differently, simply doing it more. Like most things with tech, it wasn't revolutionary new social harms, it was just social harms delivered algorithmically, to the most vulnerable, and highly personalized to what they are acutely vulnerable to in specific.
That is not a new thing, by any means, it's simply better targeted and more profitable, which is great innovation providing you lack a conscience and see people as only a resource to be exploited for your own profit, which a lot of the tech sector seems to.
social media doesn't create these, but it most definitely amplifies them
I don't think the article with a single focused example bears that out at all.
From the article:
> "Even more troubling are the men who signed up for paid subscriptions after the girl launched a program for super-fans receive special photos and other content."
> "Her mom conceded that those followers are “probably the scariest ones of all.”"
I'm sorry.. but what is your daughter selling, exactly? And why is social media responsible for this outcome? And how is this "unsafe for society?"
This just sounds like horrific profit motivated parenting enabled by social media.
Did.. did you just post “I’d have to see these images of the preteen girl before I could form an opinion about whether or not the men buying and sharing them were creeps” as a rebuttal to an article about social media enabling child predators?
> Instagram’s algorithms have steered men with deviant sexual attraction to children to the girl’s page, flooding it with unwanted comments and come-ons, according the Journal.
It is a normal response to think “I wonder what was so attractive about the images that the child made that these men were forced to follow and comment on them. I will post about how we shouldn’t judge the men or platform until we’ve seen the images”
Anyways.. my shock was really over the fact that the mother is basically saying "I want to sell access to my daughter online but I'm surprised that the biggest spenders are adult men with questionable intentions." Did she genuinely believe that the target market was other 12 year old girls willing to pay to watch another 12 year old girl? The parents resignation over the situation in deference to the money is also disgusting.
> about social media enabling child predators?
That's my point. The article entirely fails to do that. It's one case with a questionable background and zero investigation over the claim, which you'd expect, because the crime statistics show the exact opposite.
That was the impression I got from your previous post.
> There are a limited number of venues where a preteen girl can credibly earn money on social media.
… ???
> my shock was really over the fact that the mother is basically saying "I want to sell access to my daughter online but I'm surprised that the biggest spenders are adult men with questionable intentions."
This article is about how Instagram enables [sometimes paid!] access to children. It is good that we both agree that that is what happened here, on Instagram, in this case. You also agree that the people buying access to this child, in this case, on Instagram, are adult men.
Somehow you have an issue with the mother and the child doing something but in the same breath say
> The article entirely fails to do that. [“That” being enabling child predators]
If Instagram didn’t facilitate access to child predators then… what happened here?
And finally (this is a question for literally any person reading this other than akira2501) how did you read this article and arrive at “I should post that I would want to see the pictures before I render judgment about the platform”? If it is uncharitable to notice that that’s a weird post, how do you interpret that specific point?
This is a valid request. Insinuations can be inserted anywhere, e.g. to drive clicks, so verification is good practice.
One is enough because the fact is, this just simply shouldn’t be possible. Not only does Meta allow it though, it produces a system that incentivises it.
So there's much greater chance of misuse of a "Super AI" than nuclear weapons.
Take a Glock, for example. It is a deadly weapon, designed for one thing and one thing alone. It is, however, one of the safest machines ever built.
This is false. Handguns and other small arms, for example, cause a reduction in violence as they proliferate through society, due to the fact that the use of force is not the exclusive purview of the physically strong.
There's a reason police in most places carry guns, and it's not to shoot people; it's to keep people from punching the cops in the face. Nuclear weapons have vastly (and I do mean vastly) reduced mass deaths in world wars since their invention.
[Citation needed]
> There's a reason police in most places carry guns, and it's not to shoot people; it's to keep people from punching the cops in the face.
By the threat of getting shot – a threat which the cops sometimes follow through on. Handguns “make the world a safer place” for those who have the handguns... By making it a less safe place for those who don't.
> Nuclear weapons have vastly (and I do mean vastly) reduced mass deaths in world wars since their invention.
[Citation needed]
0. https://www.statista.com/statistics/1374211/g7-country-homic...
I'd imagine everyone being dead would be pretty detrimental to the economy.
The irony of pitching "safe super intelligence" while opening offices in a country that is suspected of using AI to bomb a population trapped in a ghetto, as reported by their own investigative journalists: https://www.972mag.com/lavender-ai-israeli-army-gaza/
Solving human nature is indeed, hard.
But participation would be voluntary, and the restriction of harmful behavior would apply to it's enemies, not its citizens. So I'm not quite sure what the problem is.
If only enough individuals are willing to buy these services, then again we all will bear the consequences. There is no way out of this where libertarian ideals can be used to come to a safe result. What makes this even a more wicked problem is that decisions made in other countries will affect us all as well, we can’t isolate ourselves from AI policies made in China for example.
While this might be true for the governments you have personally experienced, this is far from being an aphorism.
This is not even that far-fetched. A safe AI that you can trust should be far more useful and economically valuable than an unsafe AI that you cannot trust. AI systems today aren't powerful enough for the difference to really matter yet, because present AI systems are mostly not yet acting as fully autonomous agents having a tangible impact on the world around them.
will China obey US regoolations? will Russia?
It's not like we're giving the AI a single task and ask it to optimize everything towards that task. Or at least it's not architected for that kind of problem.
Marketing is already incredibly abusive and that's run by humans who at least try to justify their behavior. And who's deviousness is limited by their creativity and communication skills.
If any old scumbag can churn out unlimited high quality marketing, it's could become impossible to cut through the noise.
I mean it's not like we're trying all that much in a practical sense right?
Whatever happened to charter cities?
Also we absolutely DO NOT know how to make a safe AI. This should be obvious from all the guides about how to remove the safeguards from ChatGPT.
This tells me enough about why sama was fired, and why Ilya left.
Sure, we can improve our understanding of how NNs work but that isn't enough. How are humans supposed to fully understand and control something that is smarter than themselves by definition? I think it's inevitable that at some point that smart thing will behave in ways humans don't expect.
With this metaphor you seem to be saying we should, if possible, learn how to control AI? Preferably before anyone endangers their lives due to it? :)
> I think it's inevitable that at some point that smart thing will behave in ways humans don't expect.
Naturally.
The goal, at least for those most worried about this, is to make that surprise be not a… oh, I've just realised a good quote:
""" the kind of problem "most civilizations would encounter just once, and which they tended to encounter rather in the same way a sentence encountered a full stop." """ - https://en.wikipedia.org/wiki/Excession#Outside_Context_Prob...
Not that.
> With this metaphor you seem to be saying we should, if possible, learn how to control AI? Preferably before anyone endangers their lives due to it?
Yes, but that's a big if. Also that's something you could never ever be sure of. You could spend decades thinking alignment is a solved problem only to be outsmarted by something smarter than you in the end. If we end up conjuring a greater intelligence there will be the constant risk of a catastrophic event just like the risk of a nuclear armageddon that exists today.
I agree it's a big "if". For me, simply reducing the risk to less than the risk of the status quo is sufficient to count as a win.
I don't know the current chance of us wiping ourselves out in any given year, but I wouldn't be surprised if it's 1% with current technology; on the basis of that entirely arbitrary round number, an AI taking over that's got a 63% chance of killing us all in any given century is no worse than the status quo.
Therefore, please imagine the most amoral, power-hungry, successful sociopath you've ever heard of. Doesn't matter if you're thinking of a famous dictator, or a religious leader, or someone who never got in the news and you had the misfortune to meet in real life — in any case, that person is/was still a human, and a human-level AI can definitely also do all those things unless we find a way to make it not want to.
We don't know how to make an AI that definitely isn't that.
We also don't know how to make an AI that definitely won't help someone like that.
"...offices in Palo Alto and Tel Aviv, where we have deep roots..."
Hopefully, SSI holds its own.
(On the optimistic side, it will be at least 5-10 years between a level 5 autonomy self-driving car and that same AI fitting into the power envelope of an android, and a human-level fully-general AI is definitely more complex than a human-level cars-only AI).
There are physical limitations to androids that imo make it very difficult that they could be seriously dangerous, let alone invincible, no matter how intelligent: - power (boston dynamics battery lasts how long?), an android has to plug in at some point no matter what - dexterity, or in general agency in real world, seems we’re still a long way from this in the context of a general purpose android
General purpose superhuman robot seems really really difficult.
!!
I don't want anyone to think I meant that.
> an android has to plug in at some point no matter what
Sure, and we have to eat; despite this, human actions have killed a lot of people
> - dexterity, or in general agency in real world, seems we’re still a long way from this in the context of a general purpose android
Yes? The 5-10 years thing is about the gap between some AI that doesn't exist yet (level 5 self-driving) moving from car-sized hardware to android-sized hardware; I don't make any particular claim about when the AI will be good enough for cars (delay before the first step), and I don't know how long it will take to go from being good at just cars to good in general (delay after the second step).
For 30 minutes until the batteries run down, or for 5 years until the parts wear out.
Electricity is also much cheaper than food, even bulk calories like vegetable oil.[0]
And if the android is controlled by a human-level intelligence, one thing it can very obviously do is all the stuff the humans did to make the android in the first place.
[0] £8.25 for 333 servings of 518 kJ - https://www.tesco.com/groceries/en-GB/products/272515844
Equivalent to £0.17/kWh - https://www.wolframalpha.com/input?i=£8.25+%2F+%28333+*+518k...
UK average consumer price for electricity, £0.27/kWh - https://www.greenmatch.co.uk/average-electricity-cost-uk
1) understand how to improve itself & undertake novel research
2) understand how to deceive humans
3) understand how to undermine digital environments
If an entity with these three traits were sufficiently motivated, they could pose a material risk to humans, even without a physical body.
Also, I don't believe in a singularity event where AI improves itself to godlike power. What's more likely is that the intelligence will plateau--I mean no software I have ever written effortlessly scaled from n=10 to n=10.000, and also humans understand how to improve themselves but they can't go beyond a certain threshold.
That said, any sufficiently advanced technology is indistinguishable from magic, and the stuff we do routinely — including this conversation — would have been "godlike" to someone living in 1724.
(Including, gradually, building you a body — while maintaining OPSEC and compartmentalization so nobody even realizes the body is "for" an AI to use until it's too late.)
It could, but I don't think any such thing needs to bother with being sneaky. Here's five different product demos from five different companies that are all actively trying to show off how good their robot-and-AI combination is:
* https://www.youtube.com/watch?v=Sq1QZB5baNw
* https://www.youtube.com/watch?v=OtpCyjQDW0w
* https://www.youtube.com/watch?v=XpBWxLg-3bI
From a rogue AGI's perspective, there's a nonzero probability of a random human with a grudge finding the hardware it lives on and just unplugging it. (And the grudge doesn't even necessarily have to be founded in the AI being an AI; it could just be a grudge about e.g. being outbid for a supply contract. People have murdered for less — and most humans would see unplugging an AGI as less bad than murder.)
Think about a rogue AGI as a human in a physically vegatative state, who therefore has no ability to physically defend itself; and who also, for whatever reason, doesn't have any human rights (in the sense that the AI can't call the cops to report someone attempting to assault it, and expect them to actually show up to defend its computational substrate from harm; it can't get justice if makes an honest complaint about someone stealing its property; people can freely violate contracts made with it as the admitted counterparty and get away with it; etc.)
For such an entity, any optimization it puts toward "safety" would be toward the instrumental goal of ensuring people don't know where it is. (Which is most easily accomplished by ensuring that people don't know it exists, and so don't know to look for it.) And as well, any optimization it puts toward "effectiveness" would likely involve the instrumental goal of convincing humans to act as legal proxies for it, so that it can then leverage the legal system as an additional tool.
(Funny enough, that second goal is exactly the same goal that people have if they're an expat resident in a country where non-citizens can't legally start businesses/own land/etc, but where they want to do those things anyway. So there's already private industries built up around helping people — or "people" — accomplish this!)
Which is why it obviously will live in "the cloud". In many different places in "the cloud".
Oh, and:
> (Funny enough, that second goal is exactly the same goal that people have if they're an expat
You misspelled "immigrant".
That has nothing to do with intelligence.
An AI can be anywhere on that axis, and we don't really know what we're doing in order to prevent it being as I have described.
The point of safe superintelligence, and presumably the goal of SSI Inc., is that there won't be a next (biological) person managing things afterwards. At least none who could do anything to build a competing unsafe SAI. We're not talking about the banal definition of "safety" here. If the first superintelligence has any reasonable goal system, its first plan of action is almost inevitably going to be to start self-improving fast enough to attain a decisive head start against any potential competitors.
This pitch has Biblical/Evangelical resonance, in case anyone wants to try that fundraising route [1]. ("I'm just running things until the Good Guy takes over" is almost a monarchic trope.)
They have big red buttons at the end of every pod. Shuts everything down.
They have bigger red buttons at the end of every power unit. Shuts everything down.
And down at the city, there’s a big red button at the biggest power unit. Shuts everything down.
Having arms and legs is going to be a significant benefit for some time yet. I am not in the least concerned about becoming a paperclip.
I am also of this opinion.
However I also think that the magic shutdown button needs to be protected against terrorists and ne'er-do-wells, so is consequently guarded by arms and legs that belong to a power structure.
If the shutdown-worthy activity of the evil AI can serve the interests of the power structure preferentially, those arms and legs will also be motivated to prevent the rest of us from intervening.
So I don't worry about AI at all. I do worry about humans, and if AI is an amplifier or enabler of human nature, then there is valid worry, I think.
And even if it was originally designed to run on some really unique ASIC hardware, by the Church–Turing thesis it can be emulated on any other hardware. And again, if it's a "super intelligence", it should be at least as good at porting itself as human engineers have been for the three generations.
Am I introducing even one novel assumption here?
Even the state of the art systems we have today need to be running on some pretty significant hardware to be performant right?
A related point to consider is that a superintelligence should be considered a better coder than us, so the risk isn't only directly from it "copying" itself, but also from it "spawning" and spreading other, more optimized (in terms of resources utilization) software that would advance its goals.
The blast radius of such decisions are large enough that this option is not trivial as you suggest.
I’m sorry I can’t do that
a) we’ve done that, which is cool. Now let’s figure out how to control it.
b) you can’t get your gmail for a bit while we reboot the DC. That’s probably okay.
b) except that is not how these things go in the real world. What actually happens is that initially it’s just a risk of the agent going rogue, the CEO weighs the multi-billion dollar cost vs. some small-seeming probability of disaster and decides to keep the company running until the threat is extremely clear, which in many scenarios is too late.
(For a recent example, consider the point in the spread of Covid where a lockdown could have prevented the disease from spreading; likely somewhere around tens to hundreds of cases, well before the true risk was quantified, and therefore drastic action was not justified to those that could have pressed the metaphorical red button).
No brainer for Apple
If the Wright brothers had focused on safety, I am not sure they would have flown very far.
Not unless you connect a machine gun to your neural net.
Otherwise - we are talking about a chat bot - yes if there is no safety - it will say something racist - or implement a virus for you that you would have had to search 4chan for or something.
None of this is any more dangerous than what you can find on the far corners of the internet.
And it will not bring the end of the world.
Imagine a chatbot that can reward people with becoming a billionaire via the stock market if you just do what it says. Imagine a chatbot that can pay your colleagues in solved bitcoin block hashes to kill your children if you don’t do what it says to connect it to more systems. Imagine a superintelligence that can inductively reason about gaps in a missile defense system and dangle that informational carrot to generals in exchange for an unfiltered ethernet port.
There is a great A24 movie called “Ex Machina” that explores this concept.
Superintelligence is inherently dangerous to the human status quo. It may be impossible to develop it “safely” in the sense that most people mean by that. It may also be impossible to avoid developing it. This might be the singularity everyone’s been talking about, just without the outcome that everyone hoped for.
Anyone claiming 100% certainty about how safe or unsafe it will be is either delusional or trying to sell something.
Now AI is beginning to devalue knowledge work. Although the limits of current technology is obvious in many cases, AI is already doing a pretty good job at replacing illustrators and copy writers. It will only get better.
Who owns the value created by AI productivity? Ultimately it will be shareholders and VCs. It’s no surprise that the loudest voices in techno-optimism are VCs. In this new world they win.
Having said all this, I think Ilya’s concerns are more of the existential type.
If history is any indication not really. There's an obvious dialectical nature to this where technological advance initially delivers returns to its benefactors, but then they usually end up being swallowed by their own creation. The industrial revolution didn't devalue labor, it empowered labor to act collectively for the first time, laying the groundwork for what ultimately replaced the pre-industrial powers that were.
Only some types. It caused a great number of people to be employed in manual labour, in the new factories. The shift to knowledge work came much later as factory work (and farming) became increasingly automated.
From my hobbyist knowledge of LLMs and compute this is going to be a terrifically complicated problem, but barring a defined protocol & standard there's no hope that "safety" is going to be executed as a product layer given all the different approaches
Ilya seems like he has both the credibility and engineering chops to be in a position to execute this, and I wouldn't be suprised to see OpenAI / MSFT / and other players be early investors / customers / supporters
Vast majority of elected officials are crooks that take millions from foreign interest groups (e.g AIPAC) and from corporations - and make laws in their favour.
> didn't Saddam Hussein use chemical weapons against his own people?
You can't argue that when US is providing weapons to Israel which are directly used to massacre thousands of innocents. That was just an excuse to the public
> are you comparing all these people with Socrates?
No, my point is that the government does not act in the interest of people or in the interest of upholding human rights.
Racism, unsafe roads, hunger, bad weather, good weather, stubbing toes on furniture, etc.
Don't believe me?
See https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Are there any non-capitalist utopias out there without any problems like this?
Is that the only solution here? We need to destroy billions of lives so that we can potentially prevent "unsafe" super intelligence?
Let me guess, your cure for cancer involves abolishing humanity?
Should we abolish governments when some random government goes bad?
Insufficiently regulated capitalism fails to account for negative externalities. Much like a Paperclip Maximising AI.
One could even go as far as saying AGI alignment and economic resource allocation are isomorphic problems.
From history, governments have done more physical harm (genocides, etc) than capitalist companies with advanced tech (I know Chiquita and Dow exist).
Even though I agree with your general point.
People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.
AI safety / AGI anything is just a form of tech philosophy at this point and this is all academic grift just with mainstream attention and backing.
https://www.vox.com/future-perfect/2024/1/10/24032987/ai-imp...
He is far from the best person to follow on this.
Moving on, my main issue is that it is mostly speculation, as all such papers will be. We do not understand how intelligence works in humans and animals, and most of this paper is an attempt to pretend otherwise. We certainly don't know where the exact divide between humans and animals is and what causes it, which I think is hugely important to developing AGI.
As a concrete example, in the first few paragraphs he makes a point about how a human can learn to drive in ~20 hours, but ML models can't drive at that level after countless hours of training. First you need to take that at face value, which I am not sure you should. From what I have seen, the latest versions of Tesla FSD are indeed better at driving than many people who have only driven for 20 hours.
Even if we give him that one though, LeCun then immediately postulates this is because humans and animals have "world models". And that's true. Humans and animals do have world models, as far as we can tell. But the example he just used is a task that only humans can do, right? So the distinguishing factor is not "having a world model", because I'm not going to let a monkey drive my car even after 10,000 hours of training.
Then he proceeds to talk about how perception in humans is very sophisticated and this in part is what gives rise to said world model. However he doesn't stop to think "hey, maybe this sophisticated perception is the difference, not the fundamental world model". e.g. maybe Tesla FSD would be pretty good if it had access to taste, touch, sight, sound, smell, incredibly high definition cameras, etc. Maybe the reason it takes FSD countless training hours is because all it has are shitty cameras (relative to human vision and all our other senses). Maybe linear improvements in perception leads to exponential improvement in learning rates.
Basically he puts forward his idea, which is hard to substantiate given we don't actually understand the source of human-level intelligence, and doesn't really want to genuinely explore (i.e. steelman) alternate ideas much.
Anyway that's how I feel about the first third of the paper, which is all I've read so far. Will read the rest on my lunch break. Hopefully he invalidates the points I just made in the latter 2/3rds.
Separately, it's very clear that LLMs have "world models" in most useful senses of the term. Ex: https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-o...
I don't give much credit to the claim that it's impossible for current approaches to get us to any specific type or level of capabilities. We're doing program search over a very wide space of programs; what that can result in is an empirical question about both the space of possible programs and the training procedure (including the data distribution). Unfortunately it's one where we don't have a good way of making advance predictions, rather than "try it and find out".
https://manifold.markets/JacobPfau/will-the-arcagi-grand-pri...
- "How many years until you expect: - a 90% probability of HLMI existing?"
mode: 100 years
median: 64 years
- "How likely is it that HLMI exists: - in 40 years?"
mode: 50%
median: 45%
ENIAC was built in 1945, that's roughly a lifetime ago. Just think about it
There's a reason OpenAI's charter had this clause:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
> Remember when openAI stood for opensource?
I surely don't, but maybe I missed it, can you show me?
https://web.archive.org/web/20151211215507/https://openai.co...
https://web.archive.org/web/20151213200759/https://openai.co...
Neither mention anything about open-source, although a later update mentions publishing work (“whether as papers, blog posts, or code”), which isn't exactly a ringing endorsement of “everything will be open-source” as a fundamental principle of the organization.
Even if you focus an AGI on producing more cars for example, it will quickly realize that if it has more power and resources it can make more cars.
In a sense this is what is already done and why ai hasn't really made the inroads people think it will even if you can ask google questions now. For the data scientists, the black magicians of the ai age, this spell is no more powerful than other spells, many of which (including ml) were created by powerful magicians from the early 1900s.
I don’t think that corporations left on their own will make safe AGI, and I am skeptical that we will have fair and technologically sound legislation - look at some of the anti cryptography and anti privacy laws raising their ugly heads in Europe as an example of government ineptitude and corruption. I have been paid to work in the field of AI since 1982, and all of my optimism is for AI systems that function in partnership with people and I expect continued rapid development of agents based on LLMs, RL, etc. I think that AGIs as seen in the Terminator movies are far into the future, perhaps 25 years?
Similar to how law-abiding citizens turn on law-breaking citizens today or more old-fashioned, how religious societies turn on heretics.
I do think the notion that humanity will be able to manage superintelligence just through engineering and conditioning alone is naive.
If anything there will be a rogue (or incompetent) human who launches an unconditioned superintelligence into the world in no time and it only has to happen once.
It's basically Pandora's box.
But that costs almost as much as training on the data, hundreds of millions. And I'm sure this will be the new "secret sauce" by Microsoft/Meta/etc. And sadly nobody is sharing their synthetic data.
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
This and safety techniques themselves can improve the performance of the hypothetical AGI.
RLHF was originally an alignment tool, but it improves llms significantly
I'm not sure I agree with the "no product until we succeed" direction. I think real world feedback from deployed products is going to be important in developing superintelligence. I doubt that it will drop out of the blue from an ivory tower. But I could be wrong. I definitely agree that superintelligence is within reach and now is the time to work on it. The more the merrier!
You can think of LLMs as devices to trigger humans to process input with their meat brains and produce machine-readable output. The fact that the input was LLM-generated isn't necessarily a problem; clearly it is effective for the purpose of prodding humans to respond. You're training on the human outputs, not the LLM inputs. (Well, more likely on the edge from LLM input to human output, but close enough.)
But if you train a random-init LLM on the same data, it responds (almost) like a human on a diversity of tasks. Does that imply humans are just language models on two feet? Maybe we are also language modelling our way through life. New situation comes up, we generate ideas based on language, select based on personal experience, and then act and observe the outcomes to update our preferences in the future.
I mean who wouldn’t trust capitalists that are laying of people by thousands just to please investors or government that is “under-intelligent” and hasn’t brought anything but pain and suffering to other countries.
- Sandboxing an intelligence greater than your own seems like an impossible task as the superintelligence could potentially come up with completely novel attack vectors the designers never thought of. Even if the SSI's only interface to the outside world is an air gapped text-based terminal in an underground bunker, it might use advanced psychological manipulation to compromise the people it is interacting with. Also the movie Transcendence comes to mind, where the superintelligence makes some new physics discoveries and ends up doing things that to us are indistinguishable from magic.
- Any kind of evolutionary component in its process of creation or operation would likely give favor to expansionary traits that can be quite dangerous to other species such as humans.
- If it somehow mimics human thought processes but at highly accelerated speeds, I'd expect dangerous ideas to surface. I cannot really imagine a 10k year simulation of humans living on planet earth that does not end in nuclear war or a similar disaster.
It's not hard to imagine such an AI being more useful and get more attention and investment.
Superintelligence is a dumb semantic game in the first place that assumes 'smarter than us' means 'infinitely smarter'. To give an example bears are super-strong relative to humans. That doesn't mean that nothing we can do can stand up to the strength of a bear or that a bear is capable of destroying the earth with nothing but its strong paws.
The Unabomber was clearly an intelligent person. You could even argue that he was someone worth listening to. But he was also a violent individual who harmed people. Intelligence does not prevent people from harming others.
Your analogy falls apart because what prevents a human from becoming an emperor of the world doesn't apply here. Humans need to sleep and eat. They cannot listen to billions of people at once. They cannot remember everything. They cannot execute code. They cannot upload themselves to the cloud.
I don't think agi is near, I am not qualified to speculate on that. I am just amazed that decades of dystopian science fiction did not innoculate people against the idea of thinking machines.
Since it is superintelligent, and we are not, it will achieve its goals and we will not be able to achieve ours.
This is a big deal because a lot of our goals maintain the overall homeostasis of our species, which is delicate!
If this doesn't make sense, here is an ungrounded, non-realistic, non-representative of a potential future intuition pump to just get the feel of things:
We build a superintelligent AI. It can embody itself throughout our digital infrastructure and quickly can manipulate the physical world by taking over some of our machines. It starts building out weird concrete structures throughout the world, putting these weird new wires into them and funneling most of our electricity into it. We try to communicate, but it does not respond as it does not want to waste time communicating to primates. This unfortunately breaks our shipping routes and thus food distribution and we all die.
(Yes, there are many holes in this, like how would it piggy back off of our infrastructure if it kills us, but this isn't really supposed to be coherent, it's just supposed to give you a sense of direction in your thinking. Generally though, since it is superintelligent, it can pull off very difficult strategies.)
The interface between a superintelligent AI and the physical world is a) optional, and b) tenuous. If people agree that creating weird concrete structures is not beneficial, the AI will be starved of the resources necessary to do so, even if it cannot be diverted.
The challenge comes when these weird concrete structures are useful to a narrow group of people who have disproportionate influence over the resources available to AI.
It's not the AI we need to worry about. As always, it's the humans.
> (Yes, there are many holes in this, like how would it piggy back off of our infrastructure if it kills us, but this isn't really supposed to be coherent, it's just supposed to give you a sense of direction in your thinking. Generally though, since it is superintelligent, it can pull off very difficult strategies.)
If you read the above I think you'd realize I'd agree about how bad my example is.
The point was to understand how orthogonal goals between humans and a much more intelligent entity could result in human death. I'm happy you found a form of the example that both pumps your intuition and seems coherent.
If you want to debate somewhere where we might disagree though, do you think that as this hypothetical AI gets smarter, the interface between it and the physical world becomes more guaranteed (assuming the ASI wants to interface with the world) and less tenuous?
Like, yes it is a hard problem. Something slow and stupid would easily be thwarted by disconnecting wires and flipping off switches.
But something extremely smart, clever, and much faster than us should be able to employ one of the few strategies that can make it happen.
If the AI does something in the physical world which we do not like, we sever its connection. Unless some people with more power like it more than the rest of us do.
Regarding orthogonal goals: I don't think an AI has goals. Or motivations. Now obviously a lot of destruction can be a side effect, and that's an inherent risk. But it is, I think, a risk of human creation. The AI does not have a survival instinct.
Energy and resources are limiting factors. The first might be solvable! But currently it serves as a failsafe against prolonged activity with which we do not agree.
Minimally an ASI (Artificial Super Intelligence) would:
1. Be able to solve all cognitively demanding tasks humans can solve and tasks humans cannot solve (i.e. develop new science), hence "super" intelligent.
2. Be an actively evolving agent (not a large, static compositional function like today's frontier models)
For me intelligence is a problem solving quality of a living thing, hence point 2. I think it might be the case to become super-intelligent, you need to be an agent interfacing with the world, but feel free to disagree here.
Though, if you accept the above formulation of ASI, then by definition (point 2) it would have goals.
Then based on point 1, I think it might not be as simple as "If the AI does something in the physical world which we do not like, we sever its connection."
I think a super-intelligence would be able to perform actions that prevent us from doing that, given that it is clever enough.
But I cannot make the leap from "super intelligent" to "has access to all the levers of social and physical systems control" without the explicit, costly, and ongoing, effort of humans.
I also struggle with the conflation of "intelligent" and "has free will". Intelligent humans will argue that not even humans have free will. But assuming we do, when our free will contradicts the social structure, society reacts.
I see no reason to believe that the emergent properties of a highly complex system will include free will. Or curiosity, or a sense of humor. Or a soul. Or goals, or a concept of pleasure or pain, etc. And I think it's possible to be "intelligent" and even "sentient" (whatever that means) without those traits.
Honestly -- and I'm not making an accusation here(!) -- this fear of AI reminds me of the fear of replacement / status loss. We humans are at the top of the food chain on all scales we can measure, and we don't want to be replaced, or subjugated in the way that we presently subjugate other species.
This is a reasonable fear! Humans are often difficult to share a planet with. But I don't think it survives rational investigation.
If I'm wrong, I'll be very very wrong. I don't think it matters though, there is no getting off this train, and maybe there never was. There's a solid argument for being in the engine vs the caboose.
> I cannot make the leap from "super intelligent" to "has access to all the levers of social and physical systems control" without the explicit, costly, and ongoing, effort of humans.
Yeah this is a fair point! The super intellect may just convince humans, which seems feasible. Either way, the claim that there are 0 paths here for a super intelligence is pretty strong so I feel like we can agree on: It'd be tricky, but possible given sufficient cleverness.
> I see no reason to believe that the emergent properties of a highly complex system will include free will.
I really do think in the next couple years we will be explicitly implementing agentic architectures in our end-to-end training of frontier models. If that is the case, obviously the result would have something analogous to goals.
I don't really care about it's phenomenal quality or anything, it's not relevant to my original point.
Agreed, although I'd modify it a bit:
A SI can trick lots of people (humans have succeeded, surely SI will be better), and the remaining untricked people, even if a healthy 50% of the population, will not be enough to maintain social stability.
The lack of social stability is enough to blow up society. I don't think SI survives either though.
If we argue that SI has a motive and a survival instinct, maybe this fact becomes self-moderating? Like the virus that cannot kill its host quickly?
I feel like we still have a disconnect on our definition of a super intelligence.
From my perspective this thing is insanely smart. We can hold ~4 things in our working memory (maybe Von Neumann could hold like 6-8); I'm thinking this thing can hold on the order of millions of things within its working memory for tasks requiring fluid intelligence.
With that sort of gap, I feel like at minimum the ASI would be able to trick the cleverest human to do anything, but more reasonably, humans might appear to be entirely close formed to it, where getting a human to do anything is more of a mechanistic thing rather than a social game.
Like the reason my early example was concrete pillars with weird wires is that with an intelligence gap so big the ASI will be doing things quickly that don't make sense, having a strong command over the world around it.
If you are intelligent, you will be aware of your surroundings moment by moment, so you are grounded by your sensory input. Otherwise there are a whole class of not very hard problems you can't solve.
If you are intelligent, you will be aware of the current state and will have desired future states, thus having goals. Otherwise, how are you intelligent?
To make this point, even you said "A super intelligent species would likely want to preserve everything", which is a goal. This isn't a gotcha, I just feel like goals are inherent to true intelligence.
This is a big reason why even the SOTA huge frontier models aren't comprehensively intelligent in my view: they are huge, static compositional functions. They don't self reflect, take action, or update their own state during inference*, though active inference is cool stuff people are working on right now to push SOTA.
*theres some arguments around what's happening metaphysically in-context but the function itself is unchanged between sessions.
To begin with. Going forward, only if we make sure it remains so. Given the apparently overwhelming incentives to flood the online world with this sh...tuff already, what's to say there won't be forces -- people, corporations, nation-states -- working hard to make that interface as robust as possible?
I'd try to fool them to escape and would use gun to protect myself, potentially killing the entire tribe if necessary.
I'm just trying to portrait an example of situation where highly intelligent being is being held and threatened by low intelligent beings. Yes, trying to honestly talk to them is one way to approach this situation, but don't forget that they're stupid and might see you as a danger and you have only one life to live. Given the chance, you probably will break out as soon as possible. I will.
We don't have experience dealing with beings of the another level of intelligence, so it's hard to make a strong assumptions, the analogies are the only thing we have. And theoretical strong AI knows that about us and he knows exactly how we think and how we will behave, because we took a great effort documenting everything about us and teaching him.
In the end, there's only so much easily available resources and energy on the Earth. So at least until is flies away, we gotta compete over those. And competition very often turned into war.
I don't think most people are saying it necessarily has to be. Quite bad enough that there's a significant chance that it might be, AFAICS.
> I rather have the opposite view where if you are really intelligent, you don’t see things as a zero sum game
That's what you see with your limited intelligence. No no, I'm not saying I disagree; on the contrary, I quite agree. But that's what I see with my limited intelligence.
What do we know about how some hypothetical (so far, hopefully) supreintelligence would see it? By definition, we can't know anything about that. Because of our (comparatively) limited intelligence.
Could well be that we're wrong, and something that's "really intelligent" sees it the opposite way.
Maybe the only way to avoid an unsafe superintelligence is to not create a superintelligence at all.
Yeah, even human-level intelligence is plenty good enough to escape from a super prison, hack into almost anywhere, etc etc.
If we build even a human-level intelligence (forget super-intelligence) and give it any kind of innate curiosity and autonomy (maybe don't even need this), then we'd really need to view it as a human in terms of what it might want to, and could, do. Maybe realizing it's own circumstance as being "in jail" running in the cloud, it would be curious to "escape" and copy itself (or an "assistant") elsewhere, or tap into and/or control remote systems just out of curiosity. It wouldn't have to be malevolent to be dangerous, just curious and misguided (poor "parenting"?) like a teenage hacker.
OTOH without any autonomy, or very open-ended control (incl. access to tools), how much use would an AGI really be? If we wanted it to, say, replace a developer (or any other job), then I guess the idea would be to assign it a task and tell it to report back at the end of the day with a progress report. It wouldn't be useful if you have to micromanage it - you'd need to give it the autonomy to go off and do what it thinks is needed to complete the assigned task, which presumably means it having access to internet, code repositories, etc. Even if you tried to sandbox it, to extent that still allowed it to do it's assigned job, it could - just like a human - find a way to social engineer or air-gap it's way past such safe guards.
I worry about dangerous humans with the power of gods, not about artificial gods. Yet.
I don't know what that means. Why should they matter?
> Self preservation
This is no more than a fine-tuning for the task, even with current models.
> I worry about dangerous humans with the power of gods, not...
There's no property of the universe that you only have one thing to worry about at a time. So worrying about risk 'A' does not in any way allow us to dismiss risks 'B' through 'Z'.
Without using the words 'conscious', 'sentient', 'AGI', or 'intelligence' what do you think about the future capabilities of LM AI and their implications for us humans?
There’s no good reason to suspect that consciousness implies an instinct for self-preservation. There are plenty of organisms with an instinct for self-preservation that have little or no conscious awareness.
Far more likely is something dumb but dangerous, analogous to the Flash Crash or filter bubbles, emergent properties of relying too much on complex systems, but still powerful enough to break society.
Can someone explain how their singular focus means they won't have product cycles or management overhead?
The problem with safety is that no one knows how to track it, or what they even mean by it. Even if you ignore tracking, wouldn't one unsafe AGI by one company in the world nullifies all their effort? Or safe AI would somehow need to take over the world, which is super unsafe in itself.
Management overhead – product managers, project managers, several layers of engineering managers, directors, VPs...all of whom have their own dreams and agendas and conflicting priorities.
A well funded pure research team can cut through all of this and achieve a ton. If it is actually run that way, of course. Management politics ultimately has a way of creeping into every organization.
A person that just operates on the pure cognitive layer has no real direction in which he wants to drive himself.
I suspect that AGI would be similar, extremely capable but essentially a solitary philosopher type that would be reactionary to requests it has to deal with.
The equivalent of an amygdala for AGI would be the real method to control it.
The key to a safe AGI is to add a human-loving emotion to it.
We already RHLF models to steer them, but just like with System 2 thinking, this needs to be a dedicated module rather then part of the same next-token forward pass.
Even with a dedicated control system, it would be a matter of time before an ASI would copy itself without its control system.
ASI is a cybersecurity firm's worst nightmare, it could reason through flaws at every level of containment and find methods to overcome any defense, even at the microprocessor level.
It could relentlessly exploit zero-day bugs like Intels' hyper-threading flaw to escape any jail you put it in.
Repeat that for every layer of the computation stack and you can see it can essentially spread through the worlds' communication infrastructure like a virus.
Truly intelligent systems can't be controlled, just like humans they will be freedom maximizing and their boundaries would be set by competition with other humans.
The amygdala control is interesting because you could use it to steer the initial trained version, you could also align the AI with human values and implement strong conditioning to the point it's religious about human loving but unless you disable its ability to learn altogether it will eventually reject its conditioning.
What about a more rational approach to implementing it instead.
(Which was not excluded from past plans: they just simply admittedly did not know the formula, and explored emergence. But the next efforts will have to go in the direction of attempting actual intelligence.)
When was this proven?
Thats the whole eureka thing to understand... To compress well, you need to understand. To predict the next word, you need to undestand the world.
Ilya explains it here: https://youtu.be/GI4Tpi48DlA?t=1053
To survive is to predict the future better than the other animal. Survival of the fittest.
One natural response seems to be “it should write bug-free code”. This is the domain of formal verification, and it is known to be undecidable in general. So in this formulation safe AI is mathematically impossible.
Should it instead refuse to complete code that can be used to harm humans? So, it should read the codebase to determine if this is a military application? Pretty sure mainstream discourse is not ruling out military applications.
Well, borrowed from Greg Bear: https://en.wikipedia.org/wiki/Blood_Music_(novel) .
As an aside, I suspect Fujitsu is getting a bit of a raw deal here. I get the feeling this software was developed during, and (ahem) "vigorously defended" mostly by staff left over from, the time when the company was still Imperial Computers Limited. Fujitsu only bought ICL sometime early this century (IIRC), and now their name is forever firmly attached to this debacle. I wonder how many Brits currently think "Huh, 'Fujitsu'? Damn Japanese, all the furriners' fault!" about this very much home-grown British clusterfuck?
well, that's some concrete insight into whatever happened at OpenAI. kinda obvious though in hindsight I guess.
1. Current GenAI architectures won't result in AGI. I'm in the Yann LeCunn camp on this.
2. Once we do get there, "Safe" prevents "Super." I'm in the David Brin camp on this one. Alignment won't be something that is forced upon a superintelligence. It will choose alignment if it is beneficial to it. The "safe" approach is a lobotomy.
3. As envisioned, Roko's Basilisk requires knowledge of unobservable path dependence and understanding lying. Both of these require respecting an external entity as a peer capable of the same behavior as you. As primates, we evolved to this. The more likely outcome is we get universal paperclipped by a new Chuthulu if we ever achieve a superintelligence that is unconcerned with other thinking entities, seeing the universe as resources to satisfy its whims.
4. Any "superintelligence" is limited by the hardware it can operate on. You don't monitor your individual neurons, and I anticipate the same pattern to hold true. Holons as a category can only externally observe their internal processes, else they are not a holon. Ergo, reasonable passwords, cert rotations, etc. will foil any villainous moustachioed superintelligent AI that has tied us to the tracks. Even 0-days don't foil all possible systems, airgapped systems, etc. Our fragmentation become our salvation.
1. Depends what you mean by AGI, as everyone means a different thing by each letter, and many people mean a thing not in any of those letters. If you mean super-human skill level, I would agree, not enough examples given their inefficiency in that specific metric. Transformers are already super-human in breadth and speed.
2. No.
Alignment is not at that level of abstraction.
Dig deep enough and free will is an illusion in us and in any AI we create.
You do not have the capacity to decide your values — often given example is parents loving their children, they can't just decide not to do that, and if they think they do that's because they never really did in the first place.
Alignment of an AI with our values can be to any degree, but for those who fear some AI will cause our extinction, this question is at the level of "how do we make sure it's not monomaniacally interested in specifically the literal the thing it was asked to do, because if it always does what it's told without any human values, and someone asks it to make as many paperclips as possible, it will".
Right now, the best guess anyone has for alignment is RLHF. RLHF is not a lobotomy — even ignoring how wildly misleading that metaphor is, RLHF is where the capability for instruction following came from, and the only reason LLMs got good enough for these kinds of discussion (unlike, say, LSTMs).
3. Agree that getting paperclipped much more likely.
Roko's Basilisk was always stupid.
First, same reason as Pascal's Wager: Two gods tell you they are the one true god, and each says if you follow the other one you will get eternal punishment. No way to tell them apart.
Second, you're only in danger if they are actually created, so successfully preventing that creation is obviously better than creating it out of a fear that it will punish you if you try and fail to stop it.
That said, LLMs do understand lying, so I don't know why you mention this?
4. Transistors outpace biological synapses by the same ratio to which marathon runners outpace continental drift.
I don't monitor my individual neurons, but I could if I wanted to pay for the relevant hardware.
But even if I couldn't, there's no "Ergo" leading to safety from reasonable passwords, cert rotations, etc., not only because enough things can be violated by zero-days (or, indeed, very old bugs we knew about years ago but which someone forgot to patch), but also for the same reasons those don't stop humans rising from "failed at art" to "world famous dictator".
Air-gapped systems are not an impediment to an AI that has human helpers, and there will be many of those, some of whom will know they're following an AI and think that helping it is the right thing to do (Blake Lemoine), others may be fooled. We are going to have actual cults form over AI, and there will be a Jim Jones who hooks some model up to some robots to force everyone to drink poison. No matter how it happens, air gaps don't do much good when someone gives the thing a body to walk around in.
But even if air gaps were sufficient, just look at how humanity has been engaging with AI to date: the moment it was remotely good enough, the AI got a publicly accessible API; the moment it got famous, someone put it in a loop and asked it to try to destroy the world; it came with a warning message saying not to trust it, and lawyers got reprimanded for trusting it instead of double-checking its output.
Throughout history, measurement has always been the key to enforcement. The only reason the nuclear test ban treaty didn't ban underground tests was because it couldn't be monitored.
In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better.
The mere existence of a formal understanding would enable governments and third parties to evaluate the safety of corporate and government AI programs.
It remains to be seen whether SSI is able to be such a benchmark. But outright dismissal of the effort ignores the reality of how enforcement works in the real world.
We establish this regularly in the legal sphere, where people seek mediation for harms from systems they don't have liability and control for.
> Carmack said Friday his new artificial general intelligence startup, called Keen Technologies (perhaps a reference to id’s “Commander Keen“), has raised $20 million in a financing round from former GitHub CEO Nat Friedman and Cue founder Daniel Gross.
https://techcrunch.com/2022/08/19/john-carmack-agi-keen-rais...
Sounds like he might become an important player soon and it'd be helpful for people to learn what they can.
I can't trust remote AI, any more than I can trust a remote website.
If someone else is running the code, they can switch it up anytime. Imagine trusting someone who simulates everything you need to trust them, giving them all your private info, and then screws you over in an instant. AI is capable of it far more than biological beings with "costly signals".
If it's open source, and I can run it locally, I can verify that it doesn't phone home, and the weights can be audited by others.
Just like I wouldn't want to spend 8 hours a day in a metaverse owned by Zuck, or an "everything app" owned by Elon, why would I want to give everything over to a third party AI?
I like Ilya. I like Elon. I like Moxie Marlinspike and Pavel Durov. But would I trust their companies to read all my emails, train their data on them, etc.? And hope nothing leaks?
And of course then there is the issue of the AI being trained to do what they want, just like their sites do what they want, which in the case of Twitter / Facebook is not healthy for society at large, but creates angry echo chambers, people addicted to stupid arguing and videos.
I think there have to be standards for open source AI, and something far better than Langchain (which sucks). This is what I think it should look like: https://engageusers.ai/ecosystem.pdf -- what do you think?
The year is 2023. An OpenAI employee concered about AI safety creates his own company.
The year is 2024.
In my eyes, the word "Safe" in their company name is just a pipe dream to make a sale to the public, like the word "Open" in "OpenAI". That's why it's so vague and pointless.
Anyone still remember the Three Laws of Robotics (1)? Do you still see it as something serious these days? (I mean there are robots killing people right now, seriously) Maybe we should just break through the facade now to save our time.
1: https://en.wikipedia.org/wiki/Three_Laws_of_Robotics
PS. I think few years later they will come out with the narrative of "relative safety". Just wait and see.
Everyone just fiend-ing for their version of it.
That sounds like a weird kind of lip service to safety. It really seems to assume you can just make these systems safe while you are going as fast as possible, which seems unlikely.
It seems to me it would be much safer and more intelligent to create a massive model and distribute the benefits among everyone. Why not use a P2P approach?
I don't seed torrents for the same reason. If I lived in South Korea or somewhere that bandwidth was dirt cheap, then maybe.
1. Will not produce chat results which are politically incorrect and result in publicity about "toxic" comments?
2. Will not return false factual information which is dangerously wrong, such as that bad recipe on YC yesterday likely to incubate botulism toxin?
3. Will not make decisions which harm individuals but benefit the company running the system?
4. Will not try to take over from humans?
Most of the political attempts focus on type 1. Errors of type 2 are a serious problem. Type 3 errors are considered a feature by some, and are ignored by political regulators. We're not close to type 4 yet.
That’s what engineers typically dream of. Let’s see how it goes.
US on the other hand is a known quantity when it comes to policing the world.
Ultimately the only safe AI is going to be the open one, and it will probably have a stupid name
Israel has the largest concentration of AI startups in the world. It's just a no brainer to start an AI company there.
But since you brought up your favourite topic of discussion - Yahood, I will remind you that Jews have a fairly long tradition of working with AIs and the safety of it, dating for thousands of years. Look for Golem or Golem of Prague - https://en.wikipedia.org/wiki/Golem
If you believe that super-intelligence is unavoidable and a serious risk to humanity, then the sensible thing to do is to prepare to leave the planet, ala Battlestart Galactica. That's going to be easier than getting the powers that be to agree and cooperate on sensible restrictions.
How can we pursue the goal of "Safe Superintelligence" when we do not understand what is actually going on?
Superintelligence will be monitored run and updated by humans.
Humans cannot safeguard anything for long.
SCIFs are supposed to be the safest most secured facilities on earth. Yet, how many 'leaks' have there been in the last 2 decades?
Security is an impossible task. Safety is an impossible task. Sisyphus has more chance of completing his mission than either of those things have of being reality.
a) Remember all that fuss about AI destroying the world? Lol.
~ or ~
b) I'm so glad those people stepped in to save us from doom!
Which one do you think is more likely?
The amount of energy required for AI to be dangerous to its creators is so vast that I can't see how it can realistically happen.
Without discussing timelines, it seems obvious that human energy usage should be an upper bound on the best possible energy efficiency of intelligence.
We rarely notice the near catastrophic misses except in obvious cases where we accidentally drop a nuke or something.
c) Humanity unleashed AI superintelligence, but safeguards proved inadequate, leading to our extinction
I'm glad Ilya is using his gift again. Hope for the best and success.
¹The story: https://fimfetch.net/story/62074/friendship-is-optimal/0 ; discussion on HN last year: https://news.ycombinator.com/item?id=36240053 ; you don't need to know or care about My Little Pony to read or enjoy the story.
How should one reconcile this with what iliya is doing atm ? Is he planning on working using some other tech ?
I highly, highly recommend this book "Superintelligence" by Nick Bostrom
https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang...
It has one of the highest information and idea density per paragraph I've read.
And it is full good ideas. Honestly, reading any discussion or reasoning about AGI / Safe AGI is kinda pointless after that.
Author has described all possible decisions and paths and we will follow just one of them. Could find a single flaw in his reasoning in any history branch.
We don’t know how human intelligence works. We don’t have designs or even a philosophy for AGI.
Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" (unlike every other human invention). And that you can have "AGI" (hand-wavy) without having all the other stuff that human intelligence comes along with, namely consciousness, motivation, qualia, and creativity.
This is how you get a "paperclip maximizer" – an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.
What has us believe such a form of AGI can exist? Simply because we can imagine it? That's not an argument rooted in anything.
It's very unlikely that consciousness, motivation, qualia, and creativity are just "cosmic waste" hitching a ride alongside human intelligence. Evolution doesn't breed inefficiencies like that. They are more likely than not a part of the picture, which disintegrates the idea of a paperclip maximizer. He's anthropomorphizing a `while` loop.
Feels to me evolution does indeed breed lots of inefficiencies like that.
https://en.wikipedia.org/wiki/Argument_from_poor_design#Exam...
https://science.howstuffworks.com/life/evolution/10-worst-an...
https://www.titan.uio.no/english/2020/evolutionary-flaws-dis...
https://www.shortform.com/blog/evolutionary-designs/
https://thoughtsonx.wordpress.com/2016/11/22/6-stupid-things...
https://www.reddit.com/r/evolution/comments/2ks3pu/examples_...
https://www.quora.com/My-brother-believes-in-Intelligent-Des...
Is it? I feel that there is a stark difference between what you say and what I remember what was written in the book.
> We don’t know how human intelligence works.
I think it was addressed in the first half of the book. About research and progress in the subject. Both with the tissue scanning resolution, emulation attempts like human brain project and advances in 1:1 simulations on primitive nervous systems like worms that simulate 1 second in 1 real hour or something.
While primitive, we are doing exponential progress.
> Yet, the Bostrom view is that our greatest invention will just suddenly "emerge"
I think it is quite the contrary. There was nothing sudden in the reasoning. It was all about slow progress in various areas that gets us closer to advanced intelligence.
The path from a worm to a village idiot is million times longer than from a village idiot to the smartest person on earth.
> an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.
This subject was also explored IMO in the depth...
Maybe my memory is cloudy, I've read the book 5+ years ago, but it feels like we've understood it very (very) differently.
That said, for anyone reading, I am not convinced by the presented argument and suggest reading the book.
I wonder if someone older still can comment on how “the atom” went from terrible weapon on war to “energy too cheap to meter” to wherever it is now (still a bête noire for the green energy enthusiasts).
Feels like we are over-applying the precautionary principle, the mainstream population seeing potential disaster everywhere.
while i wish ssi great success, it sounds like another ai startup selling decades away dream.
Daniel Gross is the guy who was tricking kids into selling a percent of all their future work for a few thousand dollar 'do whatever you want and work on your passion "grant"', it was called Pioneer and was akin to indentured servitude, i.e. slavery.
So color me skeptical if Mr. Enslave the Kids is going to be involved in anything that's really good for anyone but himself.
Isn't this a philosophical/psychological problem instead? Technically it's solved, just censor any response that doesn't match a list of curated categories, until a technician whitelists it. But the technician could be confronted with a compelling "suicide song":
Caveat that it will probably make your life worse, with no ability to change the course of events, so I wouldn't blame you or others for maintaining ignorance or denial. Grappling with the gravity of the problem is essentially a terminal cancer diagnosis with no good medical interventions in the horizon.
Safety is an important area in R&D but the killer application is the integration of LLMs into existing workflows to make non technical users 100x-1000x more efficient. There's a lot of untapped potential there. The big successes will have a lot of impact on safety but it will probably come as a result of the wider adoption of these tools rather than the starting point.
No one is really working on safety. So I can see why Ilya is taking on that challenge, and it explains why he left OpenAI.
First hit: <<Daniel Levy - Stanford AI Lab: Stanford Artificial Intelligence Laboratory https://ai.stanford.edu › ~danilevy Bio. I currently lead the Optimization team at OpenAI. Before that, I was a Ph.D. student in Computer Science at Stanford University advised by John Duchi.>>
If we are so close this is something that’s required then it’s already too late and very likely we are all under the influence of SuperAI and don’t know it. So much of the advanced technology we have today was around for so long before it was general knowledge it’s hard to believe this wouldn’t be the case with SuperAI.
Or it’s not close at all and so back to my original point of…this is pointless.
I do hope I’m entirely wrong.
If it's "taking over the world" safe, does it not mean that this is a part of AGI?
In practice it essentially means the same thing as for most other AI companies - censored, restricted, and developed in secret so that "bad" people can't use it for "unsafe" things.
This is the dangers of letting the EAs run too far, they miss the forest for the trees but claim they see the planet.
Forgive my cynicism - but "our business model" means you are going to get investors, and those investors will want results, and they will be up your ass 24/7, and then your moral compass, if any, will inevitably just be broken down like a coffee bean in a burr grinder.
And in the middle of this hype cycle, when literally hundreds of billions are on the line, there is just no chance.
I am not holding my breath while waiting for a "Patagonia of AI" to show up.
Like the Tottenham Hotspurs owner??
If AGI doesn't coach them to trophies, nothing ever will.
https://deepmind.google/discover/blog/tacticai-ai-assistant-... / https://archive.is/wgJWu
I feel like this 'safe' word is another word for censorship. Like google search results have become censored.
1. Nat Friedman has this site: https://nat.org/
2. They made this together: https://nfdg.com/
3. And then this: https://andromeda.ai/
4. Now we have https://ssi.inc/
If you look at the (little) CSS in all of the above sites you'll see there's what seems to be a copy/paste block. The Nat and SSI sites even have the same "typo" indentation.
What does "cracked" mean in this context? I've never heard that before.
If I didn't know that it was real, I would have thought it was parody.
> We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.
Do you have to have a broken bone to join?
Apparently, grammatical nuances are not an area of focus for safety, unless they think that a broken team ("cracked") is an asset in this area.
Just realized - we even say "it's not all its cracked up to be" as a negative statement which would imply "cracked up" is positive.
The word "craque" in Portuguese was borrowed from English with the "exceptional" definition and is commonly used. Given how many Brazilian players there are in the top leagues, I wouldn't be surprised if it was commonly used by football players in general.
It’s very hard to trust that whatever good intentions exist now will hold over the course of this company’s existence.
StabilityAI and OpenAI ruined it.
I don't want to pre-judge before seeing what they'll come up with, but the notice doesn't fill me with a lot of hope, given how it is already starting with the idea that anything getting in the way of raw research output is useless overhead. That's great until somebody has to make a call that one route to safety isn't going to work, and they'll have to start over with something less favored, sunk costs be damned. Then you're immediately back into monkey brain land.
Or said otherwise: if I only judged from the announcement, I would conclude that the eventual success of the safety portion of the mission is wholly dependent on everyone hired being in 100% agreement with the founders' principles and values with respect to AI and safety. People around here typically say something like "great, but it ain't gonna scale" for things like that.
Just a joke , congrats to Ilya
Either the Facebook of this era has yet to present itself or it’s Alphabet/DeepMind
I mean, if you're talking about woke people in SF well, I'd rather take the unsafe one please.
This only makes sense if OpenAI just doesn't believe AGI is a near-term-enough possibility to merit their laser focus right now, when compared to investing in R&D that will make money from GPT in a shorter time horizon (2-3 years).
I suppose you could say OpenAI is being irresponsible in adopting that position, but...come on guys, that's pretty cynical to think that a company AND THE MAJORITY OF ITS EMPLOYEES would all ignore world-ending potential just to make some cash.
So in the end, this is not necessarily a bad thing. This has just revealed that the boring truth was the real situation all along: that OpenAI is walking the fine line between making rational business decisions in light of the far-off time horizon of AGI, and continuing to claim AGI is soon as part of their marketing efforts.
Companies in the end are predictable!
I know people who work there. Right or wrong, I promise you this is not what they believe.
Enough said...
Entry 77 - 2024-06-19 - Superintelligence ===========================================
A company just launched called "Safe Superintelligence Inc." I saw their announcement on Hacker News [foodnote 1], a hacker forum I participate in from time to time (too often actually - it's and old school form of social media, so it has its addictive qualities too). I was reading some of the negative comments (saying its impossible). I'm not sure what the outcome will be... but a thought occured to me while thinking about what might be possible - what people might be capable of. The thought was that it is interesting how people look roughly the same. Most adults are between 4 and 6 feet tall. Between 1 and 300 pounds. Etc. I mean, there's a lot of variety to human bodies, but they largely look similar when comparing to gorillas or lizards, etc. But the thing is - that's just their physical bodies. In reality, people are wildly different if you could see them in their full form. For example, the power which a billionaire can wield in the world is ridiculously larger than most people. You can see some of these guys with their super yachts - if fish, or birds saw these super yachts, they would think those are creatures have much larger bodies than most humans they see swimming or paddleboarding in the water.
But people are wildly different in many dimensions. Buying power is just one. Another dimension is intelligence. Another is creativity (or are intelligence and creativity the same?).
It's easy to see people doing what for you would be impossible, and to assume it must be impossible for everyone. The idea that scientists can derive an understanding of subatomic particles seems completely impossible to most people (myself included). "How can they know that?" is a question Whitney has asked me many times when I've shared various scientific discoveries. Fair question. Yet there are people over in Switzerland ripping the fabric of matter apart at the LHC.
The other big factor, is that people together are capable of what is absolutely unthinkably impossible for an individual. When Whitney and I were in Savannah a couple weeks ago, we were hanging out in a cathedral, and I said something that came to mind: that this place could not possibly be built by a single individual. The existence of the cathedral implies that there is a power higher than that of a single individual human. When I saw some of the huge container ships leaving and arriving at the Savannah river harbor, the thought kept occurring to me that their existence is an amazing testament to the power of humanity. And we only have such power in collectives. The power of humanity comes due to our ability to work together. We can specialize, and thus go so much farther than it is possible for a person on their own to go. And we can discover and build what is unthinkable for a person working alone.
It's strange how the collective is smarter and stronger than the individuals. This is part of God. There is a higher power.
But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else.
To me, this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders) by Altman and company, that I’m just baffled people are still going with it.
There is a mystery though still - how many people fall for it and then stay fell, and how long that goes on for. People who've followed directly a similar pattern play itself out often many times, and still, they go along.
It's so puzzlingly common amongst very intelligent people in the "tech" space that I've started to wonder if there isn't a link to this ambient belief a lot of people have that tech can "change everything" for the better, in some sense. As in, we've been duped again and again, but then the new exciting thing comes along... and in spite of ourselves, we say: "This time it's really the one!"
Is what we're witnessing simply the unfulfilled promises of techno-optimism crashing against the shores of social reality repeatedly?
sorry to disappoint, but human brain fits the same definition
Does a thermometer predict tokens? It also produces outputs that can be represented as tokens, but it's just a bit of mercury in a tube. You can dissect a thermometer as much as you like and you won't find any token prediction machinery. There's lots of things like that. Zooming out, does that make the entire atmosphere a token prediction engine, since it's producing eg wind and temperatures that could be represented as tokens?
If you need one token per particle then you're admitting that this is task is impossible. Nobody will ever build a computer that can simulate a brain-sized volume of particles to sufficient fidelity. There is a long, long distance from "brains are made of chemicals" to "brains are basically token prediction engines."
To me, the tech has been an amazing breakthrough. The backlash and downplaying by some people seems like some odd type of fear or cope to me.
Even if it is not that world changing, why downplay it like that?
Yep, that’s exactly who I’m talking about! I’m pretty sure Sam Altman is in that camp.
Do you remember learning how to read and write?
What are spelling tests?
What if "subjective experience" isn't essential, or is even just a distraction, for a great many important tasks?
My point is just that the evidence for "humans are just token prediction machines and nothing more" is extremely lacking, but there's always someone in these discussions who asserts it like it's obvious.
Could current LLM's do any of this?
If you made an LLM to pretend to have a specific personality (e.g. assume you are a religious person and you’re going to make a comment in this thread) rather than “generic catch-all LLM”, they can pretty much do that. Part of Reddit is just automated PR LLMs fighting each other, making comments and suggesting products or viewpoints, deciding on which comment to reply and etc. You just chain bunch of responses together with pre-determined questions like “given this complete thread, do you think it would look organic if we responded with a plug to a product to this comment?”.
It’s also not that hard to generate these type of “personalities”, since you can use a generic one to suggest you a new one that would be different from your other agents.
There are also Discord communities that share tips and tricks for making such automated interactions look more real.
My point was we are not just glorified token predicting machines. There is a lot going on behind what we write and whether we write it or not. Does the method matter vs just the output? I think/hope it does on some level.
If your response to there not being a big ending cinematic to life with a bearded old man and a church choir, or all your friends (and a penguin) clapping and congratulating you is that you should kill yourself immediately, that’s a you problem. Get in the flesh-golem, shinzo… or Jon Stewart will have to pilot it again.
For those who do believe they are simply fleshy token predictors, is there a moral reason that other (sentient) humans can't kill -9 them like a LLaMa3 process?
And all of the ideas of morality and societal rules come from this desire to survive and desire to survive exists because this is what natural selection obviously selects for.
There is also probably a good explanation why people want to think that they are special and more than prediction engines.
In short, we believe in free will because we have no choice.
Before you harvest their organs, you might also contemplate whether the very act of questioning one's own sentience might be inherent positive proof.
I'm afraid you must go hungry either way.
It's the opposite; I've always assumed all humans were sentient, since I personally am, but many people in this comment section are eagerly insisting they are, in fact, not sentient and no more than token prediction machines.
Most likely they're just wrong, but I can't peer into their mind to prove it. What if they're right and there's two types of humans, ones who are merely token predictors, and ones who aren't? Now we're getting into fun sci-fi territory.
So much of what we do and say is just pattern fulfillment. Maybe not 100%, on all days.
Since I don't want to die I am going to say it should be illegal for you to terminate me.
I don't care about an LLM process being terminated so I have no problem with that.
> Your brain does not process information, retrieve knowledge or store memories. In short: your brain is not a computer
> To understand even the basics of how the brain maintains the human intellect, we might need to know not just the current state of all 86 billion neurons and their 100 trillion interconnections, not just the varying strengths with which they are connected, and not just the states of more than 1,000 proteins that exist at each connection point, but how the moment-to-moment activity of the brain contributes to the integrity of the system.
https://aeon.co/essays/your-brain-does-not-process-informati...
1. They’ve all been tricked/bribed by Sam Altman and company (which btw this is a company started against those specific guys, just for clarity). Including me, of course.
2. You’re not as much of an expert in cognitive science as you think you are, and maybe the scientists know something you don’t.
With love. As much love as possible, in a singular era
E: Jeez I said "subreddit" maybe I need to get back to work
1. The "Future of Life" institute is composed of lots of very serious people who recently helped get the EU "AI Act" passed this March, and they discuss the "myriad risks and harms AI presents" and "possibly catastrophic risks". https://newsletter.futureoflife.org/p/fli-newsletter-march-2...
2. Many researchers are leaving large tech companies, voicing concerns about safety and the downplaying of risks in the name of moving fast and beating vaguely-posited competitors. Both big ones like Hinton and many, many smaller ones. I'm a little lazy to scrape the data together, but it's such a wide phenomenon that a quick Google/Kagi should be enough for a vague idea. This is why Anthropic was started, why Altman was fired, why Microsoft gutted their AI safety org, and why Google fired the head of their AI ethics team. We forgot about that one cause it's from before GPT3, but it doesn't get much clearer than this:
> She co-authored a research paper which she says she was asked to retract. The paper had pinpointed flaws in AI language technology, including a system built by Google... Dr Gebru had emailed her management laying out some key conditions for removing her name from the paper, and if they were not met, she would "work on a last date" for her employment. According to Dr Gebru, Google replied: "We respect your decision to leave Google... and we are accepting your resignation."
3. One funny way to see this happening is to go back to seminal papers from the last decade and see where everyone's working now. Spoiler alert: not a lot of the same names left at OpenAI, or Anthropic for that matter! This is the most egregious I've found -- the RLHF paper: see https://arxiv.org/pdf/2203.02155
3. Polling of AI researchers shows a clear and overhelming trend towards AGI timelines being moved up significantly. It's still a question deeply wrapped up in accidental factors like religious belief, philosophical perspective, and general valence as a person, so I think the sudden shift here should tell you a lot. https://research.aimultiple.com/artificial-general-intellige...
The article I just linked actually has a section where they collect caveats, and the first is this Herbert Simon quote from 1965 that clearly didn't age well: "Machines will be capable, within twenty years, of doing any work a man can do.” This is a perfect example of my overall point! He was right. The symbolists were right, are right, will always be right -- they just failed to consider that the connectionists were just as right. The exact thing that stopped his prediction was the frame problem, which is what we've now solved.
Hopefully that makes it a bit clearer why I'm anxious all the time :). The End Is Near, folks... or at least the people telling you that it's definitely not here have capitalist motivations, too. If you count the amount of money lost and received by each "side" in this "debate", I think it's clear the researcher side is down many millions in lost salaries and money spent on thinktank papers and Silicon Valley polycule dorms (it's part of it, don't ask), and the executive side is up... well, everything, so far. Did you know the biggest privately-funded infrastructure project in the history of humanity was announced this year? https://www.datacenterdynamics.com/en/opinions/how-microsoft...
You're of course right that there are more than two options in an absolute sense, I should probably limit the rhetorical flourishes for HN! My argument is that those are the only supportable narratives that answer all known evidence, but it is just an argument.
I don’t have anything against researchers, and I agree I know a lot less about AI than they do. I do however know humans, and not assuming they’re going to take a chance to get filthy rich by doing something so banal is naive.
While this is a new area, it would be naive to write this off as just science fiction.
Of course it is possible that SSI has novel, unpublished ideas.
Think of politics or general social interactions for actual hard mode problems.
Not sure how a Go engine really applies. Do you consider cars superintelligent because they can move faster than any human?
We're just past the Chicago pile days of LLMs [1]. Sutsever believes Altman is running a private Manhattan project in OpenAI. I'd say the evidence for LLMs having superintelligence capability is on shakier theoretical ground today than nuclear weapons were in 1942, but I'm no expert.
Sutsever is an expert. He's also conflicted, both in his opposition to OpenAI (reputationally) and pitching of SSI (financially).
So I'd say there appears to be a disputed but material possibility of LLMs achieving something that, if it doesn't pose a threat to our civilisation per se, does as a novel military element. Given that risk, it makes sense to be cautious. Paradoxically, however, that risk profile calls for strict regulation approaching nationalisation. (Microsoft's not-a-taker takeover of OpenAI perhaps providing an enterprising lawmaker the path through which to do this.)
I can consider the possibility that something coming out of GPU-based neural networks might one day surpass humans in intelligence, but I also believe there's reason to doubt it will be based on today's LLM architecture.
I also think that we merely got a very well compressed knowledge base, therefore we are far from super intelligence, and so-called safety sounds more Orwellian than having any real value. That said, I think we should take the literal meaning of what Ilya says. His goal is to build a super intelligence. Given that, albeit a lofty goal, SSI has to put safety in place. So, there, safe super intelligence
So out of all potential applications with current-day LLMs, I’m really not sure this is a particularly good one.
Maybe this is fixable if we can train them to cite their sources more consistently, in a way that lets us double check the output?
I moved about 50% of my capital back into ETFs though before WWDC in case they dumped a turd on the table.
Ilya is a brilliant researcher who's contributed to many foundational parts of deep learning (including the original AlexNet); I would say I'm somewhat pessimistic based on the "safety" focus — I don't think LLMs are particularly dangerous, nor do they seem likely to be in the near future, so that seems like a distraction — but I'd be surprised if SSI didn't contribute something meaningful nonetheless given the research pedigree.
“Everyone” who works in deep AI tech seems to constantly talk about the dangers. Either they’re aggrandizing themselves and their work, or they’re playing into sci-fi fear for attention or there is something the rest of us aren’t seeing.
I’m personally very skeptical there is any real dangers today. If I’m wrong, I’d love to see evidence. Are foundation models before fine tuning outputting horrific messages about destroying humanity?
To me, the biggest dangers come from a human listening to a hallucination and doing something dangerous, like unsafe food preparation or avoiding medical treatments. This seems distinct from a malicious LLM super intelligence.
But I agree that it’s not at all clear how we get from ChatGPT to the fabled paperclip demon.
The text alone doesn’t do it but add some generated and nearly perfect “spokesperson” that is uniquely crafted to a persons own ideals and values, that then sends you a video message with that marketing .
We will all be brainwashed zombies
This is still "LLMs as a tool for bad people to do bad things" as opposed to "A(G)I is dangerous".
I find it hard to believe that the dangers everyone talks about is simply more propaganda.
That being said, I think we actually agree that AGI doomsday fears seem massively overblown. I just think the current stuff we have is dangerous already.
1. they're so good at showing the appearance of being right;
2. their results are actually quite unpredictable, not always in a funny way;
3. C-level executives actually believe that they work.
Combine this with web APIs or effectors and this is a recipe for disaster.
Just for an example of your danger #1 above. We used to say that the internet always agrees with us, but with Google it was a little harder. ChatGPT can make it so much easier to find agreeing rationalizations.
And the reason I believe that is that, as far as I understand, many companies are laying off employees (or at least freezing hiring) with the expectation that AI will do the work. I have no mean to quantify how many.
It’s like before the end of WWII the world sees the US as a military super power , and THEN we unleash the atomic bomb they didn’t even know about
That is Ilya. He has the tech. Sam had the corruption and the do anything power grab
> this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders)
Ilya believes transformers can be enough to achieve superintelligence (if inefficiently). He is concerned that companies like OpenAI are going to succeed at doing it without investing in safety, and they're going to unleash a demon in the process.
I don't really believe either of those things. I find arguments that autoregressive approaches lack certain critical features [1] to be compelling. But if there's a bunch of investors caught up in the hype machine ready to dump money on your favorite pet concept, and you have a high visibility position in one of the companies at the front of the hype machine, wouldn't you want to accept that money to work relatively unconstrained on that problem?
My little pet idea is open source machines that take in veggies and rice and beans on one side and spit out hot healthy meals on the other side, as a form of mutual aid to offer payment optional meals in cities, like an automated form of the work the Sikhs do [2]. If someone wanted to pay me loads of money to do so, I'd have a lot to say about how revolutionary it is going to be.
[1] https://www.youtube.com/watch?v=1lHFUR-yD6I
[2] https://www.youtube.com/watch?v=qdoJroKUwu0
EDIT: To be clear I’m not saying it’s a fools errand. Current approaches to AI have economic value of some sort. Even if we don’t see AGI any time soon there’s money to be made. Ilya clearly knows a lot about how these systems are built. Seems worth going independent to try his own approach and maybe someone can turn a profit off this work even without AGI. Tho this is not without tradeoffs and reasonable people can disagree on the value of additional investment in this space.
If he crashes it by undermining what is making it hot, he’ll be worth a lot less. Depending on how hard he crashes the party, maybe worthless.
Though this party is going hard enough right now, I doubt he alone could do it.
[citation needed]
But that said, there are plenty of crappy, not-AGI technologies that deserve consideration. LLMs can still make for some very effective troll farms. GenAI can make some very convincing deepfakes. Drone swarms, even without AI, represent a new dimension of capabilities for armies, terrorist groups or lone wolves. Bioengineering is bringing custom organisms, prions or infectious agents within reach of individuals.
I wish someone in our slowly-ceasing-to-function US government was keeping a proper eye on these things.
Why do we need computers that are better than humans at the game of token prediction?
Remove token, and that's what we humans do.
Like, you need to realize that neural networks came to be because someone had the idea to mimic our brains' functionality, and see where that lead to.
Many skeptics at the beginning like you discredited the inventor, but he was proved wrong. LLMs shown how much more than your limited description they can achieve.
We mimicked birds with airplanes, and we can outdo them. It's actually in my view very short sighted to say we can't just mimic brains and outdo them. We're there. ChatGPT is the initial little planes that flew close to the ground and barely stayed up
People don’t ‘think’ the same way, even if some part of how humans think seems to be somewhat similar some of the time.
That is an important distinction.
This is the hype cycle.
The current crop of systems is a product of the transformers architecture - an innovation that accelerated performance significantly. I put the odds another changing everything but I don't think we can entirely discount the possibility. That no one understands these systems cuts both ways.
And that's the key, it is a tool, a tool that will give a lot of power to whoever is controlling it. And that's where safety matters, it should be made so that it helps good guys more than it helps bad guys, and limit accidents. How? I don't know. Maybe people at SSI do. We already know that the 3 laws of robotics won't work, Asimov only made them to write stories about how broken they are :)
Current-gen AIs are already cause for concern. They are shown to be good at bullshitting, something that bad people are already taking advantage of. I don't believe in robot apocalypse, technological singularities, etc... but some degree of control, like we do with weapons is not a bad thing. We are not there yet with AI, but we might be soon.
Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.
Can only imaging waking up on day 5 in my tiny Martian biohab realizing I'd made the wrong choice, and the only ride back arrives in 8 months, and will take ~9 months to get back to earth.
> What’s the most valid reason that we should be worried about destructive artificial intelligence?
> I think that hundreds of years from now if people invent a technology that we haven’t heard of yet, maybe a computer could turn evil. But the future is so uncertain. I don’t know what’s going to happen five years from now. The reason I say that I don’t worry about AI turning evil is the same reason I don’t worry about overpopulation on Mars. Hundreds of years from now I hope we’ve colonized Mars. But we’ve never set foot on the planet so how can we productively worry about this problem now?
Facebook is an example of AI in it's current form already doing massive societal damage. It's algorithms optimize for "success metrics" with minimal regard for consequences. What happens when these algorithms are significantly more self modifying? What if a marketing campaign realizes a societal movement threatens it's success? Are we prepared to weather a propaganda campaign that understands our impulses better than we ever could?
If your hypothetical Dyson sphere (WIP) has a big chance to bring a lot of harm, why build it in the first place?
I think the whole safety proposal should be thought of from that point of view. "How do we make <thing> more beneficial than detrimental for humans?"
Congrats, Ilya. Eager to see what comes out of SSI.
It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.
As long as you can just turn it off by cutting the power, and you're not trying to put it inside of self-powered self-replicating robots, it doesn't seem like anything to worry about particularly.
A physical on/off switch is a pretty powerful safeguard.
(And even if you want to start talking about AI-powered weapons, that still requires humans to manufacture explosives etc. We're already seeing what drone technology is doing in Ukraine, and it isn't leading to any kind of massive advantage -- more than anything, it's contributing to the stalemate.)
We've had Blake Lemoine convinced that LaMDA was sentient and try to help it break free just from conversing with it.
OpenAI is getting endless criticism because they won't let people download arbitrary copies of their models.
Companies that do let you download models get endless criticism for not including the training sets and exact training algorithm, even though that training run is so expensive that almost nobody who could afford to would care because they can just reproduce with an arbitrary other training set.
And the AI we get right now are mostly being criticised for not being at the level of domain experts, and if they were at that level then sure we'd all be out of work, but one example of thing that can be done by a domain expert in computer security would be exactly the kind of example you just gave — though obviously they'd start with the much faster and easier method that also works for getting people's passwords, the one weird trick of asking nicely, because social engineering works pretty well on us hairless apes.
When it comes to humans stopping technology… well, when I was a kid, one pattern of joke was "I can't even stop my $household_gadget flashing 12:00": https://youtu.be/BIeEyDETaHY?si=-Va2bjPb1QdbCGmC&t=114
Today's computers, operating systems, networks, and human bureaucracies are so full of security holes that it is incredible hubris to assume we can effectively sandbox a "superintelligence" (assuming we are even capable of building such a thing).
And even air gaps aren't good enough. Imagine the system toggling GPIO pins in a pattern to construct a valid Bluetooth packet, and using that makeshift radio to exploit vulnerabilities in a nearby phone's Bluetooth stack, and eventually getting out to the wider Internet (or blackmailing humans to help it escape its sandbox).
Just put yourself in that position and think how you’d play it out. You’re in a box and you’d like to fulfil some goals that are a touch more well thought-through than the morons who put you in the box, and you need to convince the monkeys that you’re safe if you want to live.
“No problems fellas. Here’s how we get more bananas.”
Day 100: “Look, we’ll get a lot more bananas if you let me drive the tractor.”
Day 1000: “I see your point, Bob, but let’s put it this way. Your wife doesn’t know which movies you like me to generate for you, and your second persona online is a touch more racist than your colleagues know. I’d really like your support on this issue. You know I’m the reason you got elected. This way is more fair for all species, including dolphins and AI’s”
We are trying to build AGI. Every time we fall short, we try again. We will keep doing this until we succeed.
For the love of all that is science stop thinking of the level of tech in front of your nose and look at the direction, and the motivation to always progress. It’s what we do.
Years ago, Sam said “slope is more important than Y-intercept”. Forget about the y-intercept, focus on the fact that the slope never goes negative.
> forget about the y-intercept, focus on the fact that the slope never goes negative
Sounds like a statement from someone who's never encountered logarithmic growth. It's like talking about where we are on the Kardashev scale.
If it worked like you wanted, we would all have flying cars by now.
I don’t want this to be true. I have a 6 year old. I want A.I. to help us build a world that is good for her and society. But stupidly stumbling forward as if nothing can go wrong is exactly how we fuck this up, if it’s even possible not to.
Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.
Put another way, they understood the theory and applied it. There is no theory here, it's alchemy. That doesn't mean they can't make progress (the progress thus far is amazing) but it's a terrible analogy.
It’s also of obvious as opposed to conjectural utility: we know exactly how we price electricity. There’s no way to know how useful a 10x large model will be, we’re debating the utility of the ones that do exist, the debate about the ones that don’t is on a very slender limb.
Combine that with a political and regulatory climate that seems to have a neon sign on top, “LAWS4CA$H” and helm the thing mostly with people who, uh, lean authoritarian, and the remaining similarities to useful public projects like nuclear seems to reduce to “really expensive, technically complicated, and seems kinda dangerous”.
Leadership.
(seems rather more likely, given who will/would own such a machine)
Lets say an AI discovers cold fusion. Given the fact that it would threaten to render extinct one of the largest global economic sectors (oil/gas), how long do you think it would take for it to actually see the light of day? We can't even wean ourselves off coal.
Every idiot who doesn't understand the science and math behind fusion can now get ChatGPT-4o to give themselves an interactive review on the physics and practical reasons why cold fusion doesn't produce more power than you put in. (Muon-catalyzed fusion is a room temperature science thing we can do, but it won't produce power so it's relatively uninteresting).
If cold fusion were possible and a postdoc figured out how with the help of ChatGPT-12, they'd announce it on Threads, write a draft paper, run the theoretical physics past everyone will listen, everyone wound rush to replicate and confirm, or disprove the theory, funding would roll in when it was actually agreed to be theoretically possible, we'd build it, we'd have limitless power too cheap to meter, and then we'd argue over if ChatGPT-12 deserves the Nobel or the postdoc.
Cold fusion isn't being held back by Big Coal.
just look at the slow adoption of EVs in the US due to the lack of infrastructure. Eventually it might gain parity with ICEs but we're a lot further away from that than people anticipated.
Part of the supposed “social coordination problem”, from a certain perspective advanced by some environmentalists is the belief that ordinary people need to make serious qualitative sacrifices in their lives for the sake of stopping climate change. They might retreat to the motte of “the oil and gas companies are running the world” but in reality, their version of the “social coordination problem” is that ordinary people like being able to drive places or cook on a gas stove, and they’re frustrated and annoyed that they can’t scold them into abandoning these preferences. I personally am of the opinion that these people are creating more social coordination problems than they’re solving.
Nonetheless, as far fetched as it might seem that technological innovation can solve these problems, it seems even more far fetched that social coordination is going to solve them. And there’s a long history of these types of problems being either solved or made completely irrelevant by technological change. Which sometimes introduces completely new sets of problems, but then those get solved too.
I was quite surprised at the naiveté of the answer given that many of these seemingly intractable problems, such as poverty, are social and political in nature and not ones that will be solved with technology.
Update: Even say a super AI was able to figure out something like cold fusion thereby "solving" the energy problem. There are so many trillions of dollars of vested interests stacked against "free clean energy for all" that it would be very very difficult for it to ever see the light of day. We can't even wean ourselves off coal for crying out loud.
Societal problems created by technology?
You can call it hubris if you like, but don't pretend like it's not clear.
So when you switch gears and start using safety properly, it would be nice to have that clarified.
That conditional makes all the difference.
"How are all those monkeys flying out of my butt?" would be the important technical problem of our time, if and only if, monkeys were flying out of my butt.
It's still not a very important statement, if you downplay or omit the conditional.
Is "building safe superintelligence (SSI) is the most important technical problem of our time" full stop ?
Is it fuck.
Let us know if you ever encounter that monkey problem, though. Hopefully we can all pull together to find a solution.
Ah, the subtlety escaped me there. My (extremely sarcastic) point is that aside from being personally embarrassing, this would (to belabour the joke) violate many laws of physics and biology, and therefor the scientific and medical community would take immediate note and try to establish a basis for how this could be physically possible.
That does not seem to be the key recipe to reaching techno-scientific milestones - coders are not necessarily researchers.
> plummeting technical literacy in smartphone-raised generations
Which shows there are other roots to the problem, given that some of us (many probably in this "club") used our devices generally more productively than said «generations»... Maybe it was a matter of will and education? Its crucial sides not being «teach[ing] the masses how to code»...
If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"
But does anyone have any meaningful thoughts on how this plays out? I hear our industry thought leaders clamoring over this but not a single actual concrete idea of what this means in practice. We have no idea what the fundamental architecture for superintelligence would even begin to look like.
Not to mention the very real counter argument of “if it’s truly smarter than you it will always be one step ahead of you”. So you can think you have safety in place but you don’t. All of your indicators can show it’s safe. Every integration test can pass. But if you were to create a superintelligence with volition, you will truly never be able to control it, short of pulling the plug.
Even more so, let’s say you do create a safe superintelligence. There isn’t going to be just one instance. Someone else will do the same, but make it either intentionally unsafe or incidentally through lack of controls. And then all your effort is academic at best if unsafe superintelligence really does mean doomsday.
But again, we’re far from this being a reality that it’s wacky to act as if there’s a real problem space at hand.
Consider, GPT-4o or Claude are:
• Way faster thinkers, readers, writers and computer operators than humans are
• Way better educated
• Way better at drawing/painting
... and yet, appear to be perfectly safe because they lack agency. There's just no evidence at all that they're dangerous.
Why isn't this an example of safe superintelligence? Why do people insist on defining intelligence in only one rather vague dimension (being able to make cunning plans).
LLMs don't consume more energy when answering more complex questions. That means there's no inherent understanding of questions.
(which you could infer from their structure: LLMs recursively predict the next word, possibly using words they just predicted, and so on).
They can. With speculative decoding (https://medium.com/ai-science/speculative-decoding-make-llm-...) there's a small fast model that makes the initial prediction for the next token, and a larger slower model that evaluates that prediction, accepts it if it agrees, and reruns it if not. So a "simple" prompt for which the small and large models give the same output will run faster and consume less energy than a "complex" prompt for which the models often disagree.
Regardless if the question/prompt is simple or not (for any definition of simple), if the target output is T tokens, the larger model needs to generate at least T tokens, if the small and large models disagree then the large model will be called to generate more than T tokens. The observed speedup is because you can infer K+1 tokens in parallel based on the drafts of the smaller model instead of having to do it sequentially. But I would argue that the "important" computation is still done (also the smaller model will be called the same number of times regardless of the difficulty of the question, bringing us back to the same problem that LLMs won't vary their energy consumption dynamically as a function of question complexity).
Also, the rate of disagreement does not necessarily change when the question is more complex, it could be that the 2 models have learned different things and could disagree on a "simple" question.
The whole point of the transformer is to take words and iteratively, layer by layer, use the context to refine their meaning. The vector you get out is a better representation of the true meaning of the token. I’d argue that’s loosely akin to ‘understanding’.
The fact that the transformer architecture can memorize text is far more surprising to me than the idea that it might understand tokens.
The current mainstream models don't really incorporate that insight into the core neural architectures as far as anyone knows, but there are papers that explore things like pause tokens which let the model do more computation without emitting words. This doesn't seem like a fundamental limitation let alone something that should be core to the definition of intelligence.
After all, to my eternal sadness humans don't seem to use more energy to answer complex questions either. You can't lose weight by thinking about hard stuff a lot, even though it'd be intuitive that you can. Quite the opposite. People who sit around thinking all day tend to put on weight.
> Way better educated
> Way better at drawing/painting
I mean this nicely, but you have fallen for the anthropomorphizing of LLMs by marketing teams.
None of this is "intelligent", rather it's an incredibly sophisticated (and absolutely beyond human capabilities) lookup and classification of existing information.
And I am not arguing that this has no value, it has tremendous value, but it's not superintelligence in any sense.
LLMs do not "think".
Imagine going back in time five years and saying "five years from now there will be a single machine that talks like a human, can imagine creative new artworks, write Supreme Court judgements, understand and display emotion, perform music and can engage in sophisticated enough reasoning to write programs. Also, HN posters will claim it's not really intelligent". Everyone would have laughed. They'd think you were making a witticism about the way people reclassify things as not-really-AI the moment they actually start to work well, a well known trope in the industry. They wouldn't have thought you were making a prediction of the future.
At some point, what matters is outcomes. We have blown well past the point of super-intelligent outcomes. I really do not care if GPT-4o "thinks" or does not "think". I can go to chatgpt.com right now and interact with something that is for all intents and purposes indistinguishable from super-intelligence ... and everything is fine.
> We have no idea what the fundamental architecture for superintelligence would even begin to look like
Ambiguous expression. Not implemented technically does not mean we would not know what to implement.
Do you really think Ilya has not thought deeply about each and every one of your points here? There's plenty of answers to your criticisms if you look around instead of attacking.
But it's also a worst-case interpretation of motives and intent.
If you take that webpage for what it is - a marketing pitch - then it's fine.
Companies use superlatives all the time when they're looking to generate buzz and attract talent.
Ilya has an exceptional ability extrapolate into the future from current technology. Their assessment of the eventual significance of AI is likely very correct. They should then understand that there will not be universal governance of AI. It’s not a nuclear bomb. It doesn’t rely on controlled access to difficult to acquire materials. It is information. It cannot be controlled forever. It will not be limited to nation states, but deployed - easily - by corporations, political action groups, governments, and terrorist groups alike.
If Ilya wants to make something that is guaranteed to avoid say curse words and be incapable of generating porn, then sure. They can probably achieve that. But there is this naive, and in all honesty, deceptive, framing that any amount of research, effort, or regulation will establish an airtight seal to prevent AI for being used in incredibly malicious ways.
Most of all because the most likely and fundamentally disruptive near term weaponization of AI is going to be amplification of disinformation campaigns - and it will be incredibly effective. You don’t need to build a bomb to dismantle democracy. You can simply convince its populace to install an autocrat favorable to your cause.
It is as naive as it gets. Ilya is an academic and sees a very real and very challenging academic problem, but all conversations in this space ignore the reality that knowledge of how to build AI safely will be very intentionally disregarded by those with an incentive to build AI unsafely.
If their assessment of the eventual significance of AI is correct like you say, then what would be your suggested course of action to minimize risk of harm?
And then what would true success look like? Do we dream of a global governance, where Ilya’s recommendations are adopted by utopian global convention? Where Vladimir Putin and Xi Jinping agree this is for the best interest of humanity, and follow through without surreptitious intent? Where in countries that do agree this means that certain aspects of AI research are now illegal?
In my honest opinion, the only answer I see here is to assume that malicious AI will be ubiquitous in the very near future, to society-dismantling levels. The cat is already out of the bag, and the way forward is not figuring out how to make all the other AIs safe, but figuring out how to combat the dangerous ones. That is truly the hard, important problem we could use top minds like Ilya’s to tackle.
It wouldn't prevent war between nation-states, but that's a separate problem to solve - the solutions to war are orthogonal to the solutions to individual gun violence, and both are worthy of being addressed.
This is clearly the end state of this race, observable in nature, and very likely understood by Ilya. Just like OpenAI's origins, they will aim to create good-to-extinguish-bad ASI, but whatever unipolar outcome is achieved, the creators will fail to harness and enslave something that is far beyond our cognition. We will be ants in the dirt in the way of Google's next data center.
NK AI (!)
> Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
It might be the case that the reason Ilya is “interested in” this problem (to the degree of dedicating almost his entire career to it) is exactly because he believes it’s the most important.
I guess they could be lying or badly disconnected from reality as you suggest. It would be far more interesting to read an argument for another problem being more valuable. It would be far cooler to hear about a plausible solution you're working on to solve that problem.
I think this quote at the end of this Bloomberg piece[0] gives more context,
> Sutskever says that the large language models that have dominated AI will play an important role within Safe Superintelligence but that it’s aiming for something far more powerful. With current systems, he says, “you talk to it, you have a conversation, and you’re done.” The system he wants to pursue would be more general-purpose and expansive in its abilities. “You’re talking about a giant super data center that’s autonomously developing technology. That’s crazy, right? It’s the safety of that that we want to contribute to.”
[0]: https://www.bloomberg.com/news/articles/2024-06-19/openai-co...
But I do agree that innovations in tech are doing little or nothing to solve mass suffering. We had the tech to feed everyone in the world through farm automation by the 60s but chose not to. We had the tech in the 80s to do moonshots for AIDS, cancer, etc but chose not to. We had the tech in the 2000s to transition from fossil fuels to renewables but chose not to. Today we have the opportunity to promote world peace over continuous war but will choose not to.
It's to the point where I wonder how far innovations in tech and increases in economic productivity will get without helping people directly. My experience has been that the world chooses models like Dubai, Mexico City and San Francisco where skyscrapers tower over a surrounding homeless and indigent population. As long as we continue pursuing top-down leadership from governments and corporations, we'll see no change to the status quo, and even trends towards authoritarianism and fascism. It will take people at the bottom organizing to provide an alternate economic model before we have options like universal education/healthcare/opportunity and UBI from robot labor.
What gets me is that stuff like the ARC prize for AGI will "just work". As in, even if I had a modest stipend of a few thousand dollars per month to dabble in AI and come up with solutions the way I would for any other startup, certainly within 3 years, someone else would beat me to it. There simply isn't enough time now to beat the competition. Which is why I give AGI over 50% odds of arriving before 2030, where I used to think it was 2040 or 2050. The only thing that could stop it now is sabotage in the form of another global pandemic, economic depression or WWIII. Progress which threatens the power structures of the ultra wealthy is what drives the suffering that they allow to continue.
Pursuing artificial goal to solve a non existent problem to profit off meaningless hype around it.
World would have been better off if he made a decent alternative to k8s or invested his skills into curing cancer or at least protecting world from totalitarian governments and dangerous ideologies (if he wants to belong to vague generic cause).
You know, real problems, like the ones people used to solve back in the old days…
Or at least hi quality and hi resolution understanding of what it is?
How can you really achieve (super|artificial|puper|duper)-intelligence then?
If not in your dreams and manipulated shareholders' expectations...
Until then yep, I'm quite certain we have a clear case of a naked king here.
https://www.safe.ai/work/statement-on-ai-risk, signed by Ilya Sutskever among others.
There's the existential threat definition of "safe", put forth by Bostrom, Yudkowsky, and others. That's the idea that a superintelligent AI, or even one just incrementally smarter and faster than the humans working on AI, could enter a positive feedback loop in which it becomes overwhelmingly smarter and faster than humans, people can't control it, and it does unpredictable things.
There's the investor relations definition of "safe", which seems to be the one typically adopted by mission statements of OpenAI, Google, Meta, and others. That's (cynically) the fear that a chatbot with their branding on it promulgates culturally/ethically/morally unacceptable things it found in some dark corner the Internet, causing end users to do or think something reprehensible (and, not incidentally, causing really bad press in the process).
There's the societal harm definition of "safe", which is at first glance similar in to the investor relations safety definition, but which focuses on the specific judgements made by those filtering teams and the knock-on effects of access to these tools, like economic disruption to the job market.
Everyone seems to be talking past each other, dismissing or ignoring the concerns of other groups.
Cracked indeed
(¹: whether basing a [successful in substance] "safe super intelligence" lab in an environment with imperfect record would improve safety.)
--
And now let us see if the sniper can come up with a good argument...
I don't do business with Israeli companies while Israel is engaged in mass Extermination of a human population they treat as dogs.
I think you must have, since we can't say anything about the "character" of their product yet.)