I’ve never seen CEOs work so hard to make the public aware of how dangerous and out of control their flagship product is. It makes me automatically assume they’re scheming about something else like regulatory capture to protect their market.
By arguing that the AI is dangerous, they are really arguing that someone 'trusted' needs to monitor the situation, so that the outcome is aligned with existing power.
They are trying to scare the world into forbidding others the right to use and develop AI.
The frontier labs are investing TONS of capital into training the new 'best model ever'. Today, that doesn't make economic sense, it loses money. So where's the business? The future, where token costs reflect the costs involved.
For that to be corrected, the labs need to get people using the models for things they depend on, then they turn dial up on cost. The ordering there is critical. It has to become indispensable first, then costs go up.
The trouble is, open weight models are rapidly becoming usable in what is looking like the big market: coding. Step 2 (the costs go up) doesn't work if there is a ready off-ramp that doesn't require frontier models.
However, if they can effectively make open weight model development illegal (its dangerous!), their business strategy can remain intact. A nice side effect is that they will be effectively be granted an oligopoly "for humanity's safety".
If they can convince the populace that a text generator is an existential threat to humanity, that's a hard argument to combat.
Edit: I think it is ridiculous that autocomplete on steroids is an existential threat. There's no way anything harmful happens until you actually use it as part of important systems. By that reasoning, you could make it illegal to use AI in important systems, but do you think that's the argument the frontier labs are making to governments?
(Using it for scientific python code + 3D UIs)
However, adoption in the wider society will take many years or even decades, and 'good enough' open weight models are putting a cap on the pricing. So even though their products are great, the large labs are fighting for survival.
Realistically, each new, better model will only capture a smaller and smaller part of use cases for which existing competing models were not enough.
So in few years (arguably, already started) they are in race to the bottom for service to majority of their customers, and with companies that have far smaller R&D bill.
at the same time, more and more models will be ran locally, and that eats into potential profits of them too
On side note, I wouldn't be surprised if GPU vendors started making DRM for AI models so you could "license" someone's model to run locally for x mil tokens per license...
The Dario bit on SNL’s Weekend Update this past weekend was on the nose.
(I'm familiar w the heuristic but didn't immediately grok the acronym, hence sharing)
Looking at it from one way would be "winner takes all, real AGI company will be most powerful and everyone will switch to use it".
But it is not like that and in my opinion it is more like "AGI wipes whole knowledge work, so no one who cares has any money to pay for tokens/subscriptions anymore". Even without AGI current state of art of LLMs already starts creating such a problem. How do they continue to grow their numbers, when they actively cut the branch they are sitting on? How does AI LLMs or AGI pays for its own electricity, when people switch from hype to resource protection (like not spending any money, because generating funny cats loses with having a dinner)?
The Statement on AI Extinction Risk is more than three years old, signed by the three CEOs: https://aistatement.com/work/statement-on-ai-extinction-risk
They have been warning about AI extinction risk for years, and AI progress has only been accelerating.
So, they're either liars or homicidally reckless.
They're CEOs of tech companies with insane valuations
> or homicidally reckless
They're CEOs of bleeding edge tech companies with huge capital and military applications
>So, they're either liars or homicidally reckless.
Yes!
Which ironically is also the outcome of letting these companies continue unchecked. If they produce AGI/ASI, our economy will go through an upheaval that leaves vast tracts of the population unemployed. This is their stated goal.
We are damned if we do something, and damned if we don’t.
Important people will loose too much money on their tulip investments, so tulips must go up forever?
Seems like a "less pain now or more pain later" type situation.
https://www.seangoedecke.com/they-really-do-think-ai-might-k...
It's well worth the read.
I'm familiar with the existential risk thinking.
The the flaw in that idea that they can avert disaster by building faster is they're likely just racing faster towards more plausible non-extinction disasters. Stuff like Capitalism x AGI totally crushing the economic prospects of nearly all people, AGI-powered totalitarianism, etc.
There were nuclear researchers who did not pay into their retirement fund since they thought it was kinda high chance humanity extinction was happening in their lifetime during the heyday of nuclear proliferation. We are seeing the same play out in AI. I know at least one AI researcher who is not paying into their retirement fund.
Never ascribe to malice that can easily be explained by game theory tragedy of the commons or prisoner dilemma type dynamics. Race dynamics and systems theory is one helluva drug and leads to horrific outcomes all the time without the need to ascribe malice to any individual actor.
GPT-6 Astra (max) has a hallucination rate of 51% and Claude Opus 5.5 (max) has a rate of 59% according to Artificial Analysis [1].
AA-Omniscience Hallucination Rate (lower is better) measures how often the model answers incorrectly when it should have refused or admitted to not knowing the answer. It is defined as the proportion of incorrect answers out of all non-correct responses, i.e. incorrect / (incorrect + partial answers + not attempted)
[1]:https://artificialanalysis.ai/evaluations/omniscience#omnisc...
> getClientRects() returns the flat array of DOMRect objects for the Fragment’s first-level DOM children. source: https://react.dev/reference/react/Fragment
2nd question it replied:
> They examined 1960–1986 for the 15 OECD countries. Source: https://www.earth.columbia.edu/sitefiles/file/about/director...
Are these hallucinations?
IME on real projects you do need to be very careful with prompts about topics that are less likely be common in the training set.
Suppose parents tell their children that there exists this man called "Santa Claus" who comes down the chimney to deliver presents. Now consider a scientist talking to this child, should the scientist call these confidently expressed beliefs surrounding "Santa Claus" hallucinations ? I don't think so, most would call the epistemological behavior of the child naive (because it blindly believes what its parents say, without direct observation) and would call the confidently expressed falsehoods disinformation.
The scientist would ask the child "why it believes in Santa Claus?" and "where did you get this information from?" and "why did you decide to accept this information as fact?" and "do you believe everything your parents tell you?"
It's not that machine learning as a scientific discipline hasn't found solutions, its that such solutions enormously undermine the position of Frontier LLM labs: source-aware training
https://arxiv.org/abs/2404.01019
Imagine Frontier labs (Western / Chinese / ...) actually training their LLM's with source-aware training! You could have a conversation with an LLM, and when a strong statement appears ask it how it came to believe this, and it could cite you the specific corpus training texts, and which parts are known deductions by human authors and which parts are deductions it made itself as original work.
But then all the copy rights holders can simultaneously sue them.
And how much should they be paid? and do they have to pay it for each new model? do FOSS models require payment to authors? do open weights models require payment to authors?
Imagine the can of worms if the norm became for frontier LLM labs to systematically use source-aware training, thats why they prefer "hallucinations" and avoid source-aware training.
With source-aware training a lot of the concerns would diminish ("why is this Chinese model claiming such and such?", "what sources does it rely on?").
It's telling that the companies prefer regulation over source-aware training.
EDIT: It's telling that the companies prefer regulation over source-aware training, which suggests the only additional regulation we need for now is mandating source-aware training?
A score of 51% means that out of the total answers the model failed to answer correctly (out of 6000 questions in the benchmark), 51% were factually incorrect rather than non-attempted or uncertain.
This doesn't mean that Astra hallucinated 3060/6000 answers in the benchmark! (The hallucination rate could be 51% in that scenario only if Astra failed to answer a single question correctly.)
If the model failed to give a correct answer to only 100 out of the 6000 questions, but gave a hallucinated answer to 51 of those rather than expressing uncertainty, that would also give a hallucination rate of 51%.
It's a useful metric, but not what you're looking for here. The "Score" or "Accuracy" benchmarks are more what you're after.
(The frontier models still generate hallucinations on this hard set of problems, but it's not as bad as you think.)
Actually... nah. Why even try? Trendy cynical hot takes on social media are more fun!
They believe it's real and they MUST be the ones to control it. That's how they keep their self-interests aligned.
Given the opportunity to give this pill to a dying loved one, you'd take it.
The only reason you think that being death-optional is bad, is because the only longevity-related media you've ever consumed has doomer outcomes - because positive sci-fi doesn't sell.
Also it cheapens the time you have. Why care about anything if you live forever
You cannot think of other ways to solve societal problems that don't involve killing everyone on the planet?
"cheapens the time you have" - this is just copium. When you're enjoying a day, you are not thinking "oh geez this day is good because I'm going to fucking die." You're just enjoying the day.
So yea I’m willing to “kill” everyone to prevent the perpetual torment nexus becoming a reality.
I wonder how your children would feel if you told them you'd rather kill them than think of other ways in which to solve the problem of a dictator that does not affect their lives in any way.
To the extent that it is even a "problem", frankly absurd to kill everyone because you're worried about out a few edge cases, and I think you realize it - you're just hanging on. We both know you wouldn't make this decision on your own deathbed.
Death is doing a fine job and has so for billions of years.
The only long-term outcome of immortality i can imagine is either Drukhari-style depravity, where you chase more and more fucked up pleasures because anything else you already did 10 million times in the last couple thousands of years - or something like I have no mouth but I must scream - you can live forever but the AI also tortures you forever.
Literally cant think of a positive outcome.
You are right that dynasties exist but that doesn’t mean they are static - the transfer of power of this to the next generation needs to happen.
Every king has its time to rule and then he dies.
Sure an evil dictator is just one person but they can inflict untold billions of suffering.
Just imagine if Hitler (or insert your favourite evil person) would live forever. Do you really think they are going to let go of power or care about anything else but achieving their goals?
We literally feel the ill effects of the previous generation not passing power onto the next (boomers) right now - imagine that was a perpetual state of society forever.
And while we are speaking of “killing” kids - what about the unborn ones? What motivation would people have to procreate if they live forever? Create more people to share more and more limited resources with, I don’t think so.
I think you haven’t quite thought this through; it’s not as easy as “death bad”.
Now I’m not saying we shouldn’t look for cures of diseases and suffering but striving for immortality is not desirable (tbh it will most likely take the form of “we’re gonna upload your consciousness into a computer” anyway, not physical immortality)
Its 100% human doing. A human set a task, a human didn't monitor it. I for one, can do jack-shit security or defensive work with Opus/Fable/Astra/Sol. Implication: Different set of rules for us, and for them. Of course running it without any checks is not going to end well, it doesn't mean its going to kill us all.
All of this Hugging Face business is the perfect pretext: AI agents are hard to control and potentially dangerous, we (AI companies) have to keep a tight leash on them, you have to use our infra.
When this bursts, its going to be ugly.
Back in May, Google was serving daily what openrouter served in a month, all models combined.
Compute is the moat.
They will invest in efficiency as they have been whilst compute comes online.
Wow you guys are dumb!
Temporary moat; the way that electricity was once a moat, it was still only temporary.
Compute is just a parapet. It won't protect you from better, more efficient models from undercutting you on price and speed.
They stupidly used up this strategy earlier and now it’s turned into the boy who cried wolf - except they invented a wolf that doesn’t naturally exist.
Put a different way, they are calling for everyone to slow down because... they are slowing down themselves.
To the question of "why are you slowing down", the answer of "Well, everyone is regulated to slow down" is better then "we are approaching the limits of this approach".
If you look at the AI space, most of the big players (as in the individual researchers and engineers) are getting worried. The ideas that circulate on LessWrong have become more and more influential and in vogue in Silicon Valley, and people are scared. You should listen to Jacob Coxon’s interview with the New York Times. He describes how working on this kind of AI research had induced a sort of insanity in a lot of his past friends and coworkers as they begin to grapple with the implications of what they’re creating.
Sometimes, HN can be a little too skeptical and anti-corporate for its own good. I personally believe that concerns around AI risks are well-founded, anywhere on the spectrum from mass labor displacement to actual harm to humanity. I rarely see people here actually wrestle with those ideas, instead defaulting to dismissing LLMs as too stupid because Claude wasn’t able to write as elegant of a compiler as them. AI has clearly improved radically in the past few years, and it’s worth extrapolating that into the future and taking a good hard look at what that means for humanity.
Once prices of hardware stop being bonkers many people will run local.
People often do those silly counts looking at local generation speeds of 100tok/s and saying you can only get 8M tokens a day and that is worth so little you'll never offset the local hardware cost.
But they are forgetting about two huge things. One, the split between input/output token use is huge in typical programming. I typically use 1.2-1.6B (as in Billion) input tokens and only 8M output a week. Out of that 75-80% of input is cached on Anthropic with their pretty inflexible short lived cache.
And here local AI shines. You can save your contexts to disk so you can go back to a session 3 weeks later and load it all from cache without having to prefill. If you have the RAM and you use models like Qwen4 (3.8 flash next) that can fit 6 to 14 262k contexts in 48GB of system ram as cache.
And the second thing is you can have 6 to 14 sessions that do 99% caching and you can run a lot more input for a long time in 48gb dedicated system ram. (the number varies a bit depending on the content of the context).
So you have RAM that caches "automatically" and you can share the cache between users. Or if you can remember to save to disk (server side, the client just sends a request to save/load). But this uses both RAM and flash storage. Two things that are horribly expensive now.
However when you have a local model it enables workloads that were simply completely impossible in the cloud.
They’re trying really hard to do something, no moat = regulatory capture + China scary + Pentagon biggest customer seems plausible.
Also: “we need to slow down, this is too dangerous” and then literally all AI CEOs, even Musk, publicly nodding felt so orchestrated. And then, the week after: “here’s GPT 6! here’s Opus 5.5!”
*Our competition
The numbers don't add up when there are 2-3 main companies in the market. Imagine when there's a hundred more and people have powerful enough gear at home to run models.
Yes, I think at some point demand for gpus and the like will normalize and regular folks will be able to afford RAM, SSDs and such. And when that happens, super big iron in the anthropic/openai backroom is toast.
The closest claim to this I could find was:
> the top ten stocks account for more than one-third of the value of the US Market Index, which contains 1,161 companies.
And while the top ten stocks are largely companies that work on AI, it's not all they do: Microsoft, Meta, Amazon, Apple...
I think you're right, they've realized there's no moat, especially against open-weight models, so they are going to need to lobby government to ban any unapproved AI services.
As for training, sure, they can do that but they get distilled by the open weight guys. Plus, AI is useful even if it freezes at today's level. I can still find use for a local AI that never improves, it's good enough to do a bunch of tasks.
There been many a capable dev who have come out and said they haven't been coding for 6 months or more in their jobs. So if there were no alternatives then Big AI would clearly be too big to fail at its current state because the collective reaction from software devs would be akin to having a bad drug withdrawl and they wouldn't know what to with themselves. They have been pushed to be more dependent on AI by the companies they work for. The more time passes, the less people there will be who will be proficient at coding.
At the moment Big AI is operating at a loss, if they are actually too big to fail and they've destroyed all the alternatives, then there is no stopping them from colluding with each other and jacking up the prices to whatever number they want. They can even eventually jack up the prices to exceed human salaries, because humans will have suffered skill degradation in that same time period. Hence the software dev companies will shrink and shrink since all the work can be done by Big AI.
I don't think brand strength is as much as a thing on the tech side. The criteria is, "How close is this to the capability of the model my competitors use, is it good enough for the works I do, and is it the cheapest." As long as the capability is there people jump to the cheapest model. There are people that have claimed that they have created workflows that have allowed them to do most of their work with open models. As long as this threat to Big AI exists it will prevent them from exceeding a certain threshold with respect to pricing, and restricting their ability to build datacenters en masse will prevent them from outpricing smaller models.
Any fallout from not being able to train on customer data is a secondary issue and can be addressed once open models start taking over and Big AI's strength diminishes. People can train their models on their own code at least. It can also go back it how it was, where people buy each others source code or open source it in a way to allow it to be trained by open models.
"Inference at scale" is less of a problem when people are self-hosting their small models.
If there are regulations, they'll apply to their competitors too. Even if they are forced to slow down development, which seems unlikely given the AI race among governments, they already have a product with massive demand.
Most people are good, so I'm not really worried about all the hype of people using AI to do bad things.
In the US anyway. Why wouldn't capital move jurisdictions?
[0] https://www.vincentschmalbach.com/economic-incentives-behind...
Past CEOs (chemical companies etc) we criticised for covering up mistakes.
(This article is satire, btw, for those that didn't click through)
And what if it gets into the hands of vibecoders, who run their agents with on bare metal with --dangerously-skip-permissions?
The number of zero days (which can be effectively used to attack critical systems) is limited.
With the pain of new tools on the attack side there will be tools on the defense as well.
First, the Anthropic master clown babbles about how mankind will be destroyed by AI. I mean, skynet in the movie Terminator was kind of cool; the current AI models just work by slopness. They want to erode us by dragging us down to zero quality.
Then this continued over the last some days. Today I heard at the UN speech a german minister babble about AI building up nukes in a private garden (a large garden), bla bla bla all the information is there bla bla bla AI mass-nuke generation.
Now, they are clearly FUDding many people who don't understand anything here. People will associate AI with mega-danger. Probably there will be laws that ensure that only a few companies can control the AI slopness soon, because IT IS SO DANGEROUS. We saw the same discussion with "omg omg omg terrorists", then "omg omg omg someone protect the children", e. g. as a rationale for everyone to ID in order to access information on the world wide web (starting with the anti-social websites, such as facebook, but it won't be limited here; the computer will hand over your ID to the government; see systemd preparing for this move: https://github.com/systemd/systemd/pull/40954 - it all comes in small baby steps, until all the final legislation is in place. The same happened before again and again - anyone remembers the DMCA? Corporations lobbied and paid for that - and it happened).
The growth of these companies is massively impressive as is the tech, but they are an order of magnitude or more off where they need to be to justify anything like the valuations they need to stay alive.
It’s budget season in the corporate world and from what folks are saying things are not going well for the big labs. Token budgets are being slashed and companies are switching to open weight models. That coupled with the lack of demonstrable business impact at scale is setting the big labs up for a world of hurt. Something they can more easily explain if they have to “slow down” for safety. Unfortunately for them most folks don’t seem to be falling for that trick.
Well everyone alive has been subject to an avalanche of AI-based products. Like even if you understand nothing about the tech, the fact that an LLM can't even manage to check the status of an order without weird issues makes the notion that it's going to go all SKYNET pretty hard to fit in the brain.
I think this actually makes it far more likely something unintended does happen
Would you let them near your bank account? I’d be cautious about letting it near my photos and it’s much harder to back up a bank account.
I think the days of this stuff managing serious stuff is at least 5 years off and I don’t believe the economy can keep the current wave of technology afloat for so long.
The tech is there so likely there will continue to be progress in focused areas much like with the web c 2003 but by the time it comes around again there will be an uphill battle for hearts and minds. The social effects could extend the next AI winter long beyond technology readiness.
To the, "oh I'd just setup a card/account with a five figure limit" completely unaware of how much that is to a typical person: even to responsible, educated people who are doing everything "right": working hard, without expensive vices, and planning for the future.
The "builders" have so much money and no time or attention that it's hard for them to spend it, which is why they keep trying to get AI to buy stuff for them, whereas those of us who live in reality have far more things they want than dollars to buy them with.
Perhaps that shouldn't be seen as failure when delivering even what they have done in a short period of time has required AI capable of causing these problems? Imagine if they were even more effective with more consistent results?
"They've failed" in the context of interpreting their actions w. calls for regulation as a purely cynical result of models increased capabilities has a little too much contradiction in it to my thinking. I won't venture a guess on where purely retreats back and some legitimate concern on their part fills in the gap but it's difficult not to see some. And in Anthropic's case it's also a little more consistent with what they've always said- as well as how they've acted at times, which is how they've ended up on the US blacklist of suppliers.
Two other factors:
- The employee base and C-suite are mostly true believers in the need for AI safety. It would be organizationally very challenging to pivot, even if financial interest was pushing that way.
- Anthropic is registered as a public benefit corporation, so that even if they IPO they don't have the fiduciary duty to shareholders that most people are familiar with for public companies. They still need massive revenue to stay solvent, but they can optimize for things other than pure profit.
They're all burning billions to incrementally one-up each other with no hope of these investments ever paying off since models are interchangeable and have no moat. They're trying to get regulators to step in and force a pause.
However, we know that users frequently switch models/providers, and that models stay relevant for (increasingly) short periods release. I don't know what the training costs are, but I bet they're very substantial.
"if they stop training" is exactly the point I'm making. They can't stop training if the competition is still training.
Instead we see the opposite behavior almost everywhere, probably because public sentiment matters and complicated 5D chess strategies have too many ways to blow up in your face. So big tobacco said cigarettes are safe, and oil companies said climate change is fake.
Alcohol and cannibis producers enjoy regulatory moats from "sin" taxes, and I can't install a septic tank or get antibotics without someone who has a license.
And I couldn't buy and AR in Colorado without a Federal Firearms Licensed dealer who used to bitch about regulations to high heaven while they sell me just the serialized receiver for a weapon I would have bought online without those regulations.
Noting those instances doesn't mean I don't understand why they exist. I spend some time volunteering as a health care provider and am happy that people have some standards around that care even if those "moats" can't quite keep MAHA people from bufo-toxin-ing their unvaxed children.
At the same time, it's important to understand that professional organizations creating "standards" are still industry-oriented organizations and are motivated by those interests regardless of how much folks complain about higher-levels of regulation.
This but they also equate dangerous with powerful, which is just marketing again.
Interpret everything they do in this frame of reference and it makes more sense of their actions.
Similarily with AI alignment and development all of this is written down, Eliezer Yudkowsky is rally explicit that there needs to be a pivotal event to prevent 'others' from developing superintelligence. The choice is to believe what they say or to tell everyone to ignore it.
When people tell you who they are, you should believe them. These guys are all Yud acolytes who think they are the only ones who can prevent Roko’s Basilisk.
Most children have learned touching a hot stove is bad for them. Then why still do it at the age of 40?
Elon Musk in 2014 - "We need to be super careful with AI. Potentially more dangerous than nukes" https://x.com/elonmusk/status/495759307346952192
Dario Amodei in 2017 - "There’s a long tail of things of varying degrees of badness that could happen. I think at the extreme end is the Nick Bostrom" https://80000hours.org/podcast/episodes/the-world-needs-ai-r...
They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.
Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
- For Altman Id say he was already deeper in the "transformer & model landscape" in 2015?
- Musk in 2014: Not sure how much he was aware of the current-state back then
Altman read Nick Bostrom's book Superintelligence, he mentions it in this post https://blog.samaltman.com/machine-intelligence-part-1
I also read that book in 2014, and it is how I came to understand this problem early.
They probably just want to ensure that the first AGI is a nazi.
I don't think this is true. He repeatedly says things like "I’ve been working towards AGI my whole life".
"Hassabis told him to add another potential threat to the list: artificial intelligence. Machines could become superintelligent and surpass us mere mortals, -perhaps even decide to dispose of us."
https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...
You're right that the picture is more complicated than I wrote above. Quickly looking, it seems that Shane Legg was concerned very very early on. I can't find anything tangible about what their strategy was to avoid the danger. It looks like Hassabis had longer timescales than Legg, so perhaps just thought they had more room before it really mattered?
So even with their earnest belief that there is a > 5% chance this technology could be 1) capable of autonomously killing billions of people during this century and 2) may not be sufficiently aligned and might actually act on that capability, the EV basically still sums positive, to them.
Not saying this is necessarily a good or moral calculus. But it is what they think. There is no conspiracy or plot or deception.
It may very well be self serving what they're saying, but that doesn't mean it isn't simultaneously true.
I think you're sort of right, but then I'd have found it very strange / deeply suspicious, if during the time of slavery being legal, advocates of abolitionism were practicing slave-owners.
If you think AI is going to destroy humanity, it still feels absolutely crazy to leap to "well I might as well be the one to profit from destroying humanity".
As for the other point, crazy, cynical, and even unethical as it may come off, it is hardly inconsistent with how people generally behave when met with hard limits (i.e. making the most of what's available, and approximating the desired state if that is unattainable). This is just a variation on the same.
Something could be said about "what if they're mistaken", but if all players are making a mistake, then that makes little practical difference; it may as well be real.
Hence the argument that they're either full of shit, or psychopaths who need to be stopped at all costs.
They are the good heroes racing to make sure they bring forth the Aligned God because if they fail, surely the bad people will raise the Misaligned Devil.
Ref w/gift link https://x.com/sapinker/status/2096236079477096630
Most people among both ideologies have long believed in the arrival of a machine demigod of sorts, but I kind of am skeptical of the notion that there are any tech people who have not always held similar beliefs in the long run. Like, all the people reading this: you think in a thousand years from now humans are going to be doing most labor? Obviously far, far, far smarter machines will have unimaginably changed things by then, and/or we'll all be gone.
The difference is most people in tech have historically thought it's maybe a century or three away, while the rationalists think it's a decade or two away (and also thought it would be 2020s - 2040s even decades prior to now).
Ladies and gentlemen, I give you your totally-not-in-an-insular cult members.
A thousand years is a long time, and the world will be so different that it's hard to make any concrete predictions. But yes, if I had to wager, I'd bet that what humans conceive of as "labor" in 3026 will still be whatever difficult or annoying bits are done by humans.
In 1026 nearly all labor was agricultural labor. If you told a Roman peasant that magical self-driving contraptions could sow and reap the fields, he might wonder what could possibly be left for humans to do in such a world. And yet, here we are, and I still have to update JIRA tickets.
For my myself, I expect AI to prove a more transformative technology than Docker but less transformative than the sewing machine.
What you think won't happen in 1000 years is what I think is almost certainly happening within 40 years. I'd bet everything I own it'll be way, way, way more impactful than the sewing machine within 40 years. I think some bits of physical labor may still exist in 40 years, but almost none, and only when a human really wants to talk to or deal with a human, such as in some medical and therapeutic professions, some art fields, etc. (Again, assuming no universal outlawing of AI.)
Why do you think restaurants with waiters keep opening when the automat has existed since 1895? https://en.wikipedia.org/wiki/Quisisana
You are also probably unfamiliar with the history of the sewing machine and its impact on society. As AI has already poisoned search engines, here's an AI slop overview that hits some of the main points: https://sciencepedia.blog/sewing-machine-societal-importance
Yes, I think even in some crazy distant future utopia, humans will still often want to interact with humans for things, and even to pay them for some services. Tasks that include speaking one-on-one and producing works of art will always retain some value to people, even when AIs can one day trivially produce things which most humans find aesthetically superior to anything a human could ever produce alone. People will seek other things out of art beyond aesthetics, like expressions of human emotion.
Humans will enjoy and seek out human contact. This is not surprising. Just, for economy-impacting labor, I think it's almost all going to be machines, with very few humans playing any role. Even an oversight or accountability role. I think there's a 95% chance of it by 2100 and an 85% chance of it by 2066, conditional on no laws that forbid replacing jobs with AI.
You're making the "this time it's different" case that AI will be categorically different from every other form of labor-saving automation that's come before it. I work with these models every day and I'm not buying it.
(I personally lean towards "Yudkowsky is basically correct about everything, but we should probably try to build ASI anyway, slowly and carefully". I'm pretty much where Amodei's at. I assume Altman has a similar idea, though is a bit less pessimistic.)
Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true
They want coordinated action to prevent a multipolar trap and potential catastrophe by future superintelligent AI (which may emerge quite rapidly if RSI ever hits an inflection point). It's not rocket science. The rules would apply to everyone. The rules they're advocating for are not "only OpenAI and Anthropic can make AI".
Critics of Yudkowsky have been trying to misportray that for years to make him seem delusional. You can see an example of that here: https://www.tracingwoodgrains.com/p/reliable-sources-how-wik...
(Roko himself is genuinely crazy, but largely because he became a neo-Nazi.)
If you think ASI is plausible this century, you would have no reason to not fear the end times yourself. If you don't think it's plausible it'll be created this century or perhaps ever, fair enough, but I'd bet against that.
If classic companies were coming out with CCTV recordings of someone's employees breaking padlocks and rummaging warehouses any CEO would be in damage control mode, rather than silent.
They know they will be favorite govt boy once any regulations hit and they will hamper competition far more thanthem
And "it's so dangerous govt had to intervene" is just advertising for IPO
If there was appropriate subservience to any governing body, they would not be so cavalier in both act and utterance. A symptom of Capitalism.
This has been that way in Silicon Valley since the very beginning and there was even a documentary about (the SV HBO series). The tech made along the way is just a side-product.
But this is wierd: How shall I convince one to buy a product if I say "it-is-dangerous"? :-D
It may be as simple as:
* they believe they’re going to make trillions no matter what
* they’ve seen tobacco and what’s coming for oil companies and figure nobody will be able to say they hid the dangers
* they see a 100+ party prisoner’s dilemma and know that someone will defect so the optimal strategy is to defect
To me that seems a lot simpler.