I'm really disappointed to see the HN reaction of straight up scoffing at this like it's irreversible and so let's just step on the gas. It's some weird combination of techno-utopianism and nihilism and good old elitism.
The concerns about all the jobs going away have been persistent but the labor market continues to be extremely strong. As long as entrepreneurship is creating new business opportunities and new ways to turn labor into value, it's completely possible that the effects of automation are drowned out by new applications of labor. In fact, this is what we've seen throughout the industrial revolution: constant anxiety about automation that fails to materialize in to mass unemployment.
The concerns about polluting the information space are still hypothetical as well. The current information space is a complete garbage fire and it's not due to generative AI. Might it get worse if bots start spamming people with GPT generated text on social media? Yeah it might. But social media is already terrible and we need to do something about it regardless of that.
Then there's the people looking ahead, foreseeing a future where superintelligent AIs are more powerful than humanity, and worried that most possible variations of those superintelligences are incentivized to destroy us.
I think this open letter puts much too much emphasis on the petty stuff, I think because they're trying to appeal to people who are allergic to anything that requires extrapolating more than a little bit into the future. But buying more time for alignment research, before we tackle superintelligence, does meaningfully improve humanity's odds of survival, so I hope this happens anyways.
In previous technological revolutions, affected workers were hurt by their loss of income, and some no doubt fell into poverty without ever recovering. Not everyone can be retrained for new types of jobs immediately - (1) they may not have the needed foundational knowledge or the cognitive flexibility/ability, and (2) there might not be enough of the new types of jobs emerging quickly enough for them. Not every displaced miner can become a coder, or be competitive for junior dev jobs.
(Why should the state provide for these workers? Well, primarily for humaneness, and also social stability.)
The rewards of automation (cost savings as well as profits) are reaped by (1) the capital owners of the automation technology companies (and their higher-paid employees), as well as by (2) the companies and consumers using the new automation; therefore those owners and beneficiaries could be asked to bear at least part of the costs of supporting, retraining, and placing in jobs the workers they displaced. In a nutshell: Redistribution during structural unemployment caused by technological transitions.
A humane policy would provide the above types of support for workers displaced by automation. Ideally it would already be handled by existing unemployment policy, but in many countries such support is limited or minimal.
Corporate taxation might need some rethinking along the lines of job-displacement effects of companies (a tricky question, I admit - I've come across one or two proposals for assessing the automation level of companies for taxation purposes). The cross-border dynamics add further complexity, given that automation will displace many jobs outsourced across borders.
Given that the current AI revolution looks like it will be causing even larger and faster changes than previous revolutions, such policies are imo needed as a prerequisite (one of several) for allowing the development of powerful job-displacing AI.
So then instead of finding a solution to those issues, let's instead focus all resources on a tech which will make the issues worse...
> Ryan: OpenAI [...] It's kind of a completely open-source AI project?
> Eliezer: That was the original suicidal vision, yes. But...
> Ryan: And now they're bent on commercializing the technology, is that right?
> Eliezer: That's an improvement, but not enough of one, because they're still generating lots of noise and hype and directing more resources into the field, and storming ahead with the safety that they have instead of the safety that they need, and setting bad examples.
> [...] I would change the name to "ClosedAI". Because right now, they're making it look like being "closed" is hypocrisy. They're, like, being "closed" while keeping the name "OpenAI", and that itself makes it looks like closure is like not this thing that you do cooperatively so that humanity will not die, but instead this sleazy profit-making thing that you do while keeping the name “OpenAI”.
https://www.lesswrong.com/posts/e4pYaNt89mottpkWZ/yudkowsky-...
Admittedly, the link between improvement in loss and improvement of capabilities may break down or be misleading.
However, there's just not much training data on the internet left unused. Maybe an order of magnitude. All books ever published (in English?) are a smaller dataset than the corpora already used for training. See [2] (which includes an easy summarisation of much of [1]). And the scaling laws show training data is already the bottleneck rather than compute.
[1] DeepMind, 2022, Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556
[2] Chinchilla's wild implications https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla...
More importantly, These models are extremely sensitive to loss. 2.0 to 1.8 might not seem like much but it's a huge gain in performance.
GPT-2 loss was 2.57. GPT-3 was 2
and there is plenty of training data left. perhaps not easily accessible but it's there.
I hadn't actually looked at the LLaMA paper, that's an interesting note. However AFAICT GPT3, LLaMA and Chinchilla do not use the same tokenizer, so their losses are not comparable. GPT2 and 3 use (the same) custom BPE tokenizer. LLaMa uses SentencePiece but that generates a vocabulary specific to the training data it's run on. Chinchilla used "a slightly modified SentencePiece tokenizer that does not apply NFKC normalisation. The vocabulary is very similar– 94.15% of tokens are the same as those used for training Gopher".
Even if there is a lot more text available, it doesn't mean it's good training material. And the better free sources are already used. E.g. LLaMa was trained on 64% of GitHub that had a compatible license (and you're not going to gather much more source code than that), all the free book texts they could find, all of arXiv, all English pages in CommonCrawl that classified as "reference" quality, etc. arXiv, for example, isn't all scientific papers ever, but it's a large fraction of them. All private emails stored by a large email service would probably be one of the biggest untapped valuable sources.
The idea that "x is technically possible, therefore it is inevitable" - the favored line of the tech oligarchs who are the very ones directly furthering x - is sadly cargo culted by many rank-and-file technologists. If you were to apply the same defective reasoning to, say, nuclear weapons, then we would all be dead by now.
It is possible for humans to agree that something should not be done, and prevent it from being done. This is especially true of these LLMs, which require ingesting (and ignoring copyright on, by the way) massive amounts of human work and then spending massive amounts of computation to process.
That we could choose to do something about this should not be controversial, regardless of what the driving AI.
Whether delaying AI development a little matters depends on whether you think the success of AI alignment, applied to future superintelligence, is overdetermined to succeed, overdetermined to fail, or close to borderline. Personally I think it looks borderline, so I'm glad to see things like this.
Under what definition of easy? If it's easy to stop, then why don't the people signing the letter just do it, rather than trying to appeal to others to do it instead?
Aligning thousands of people (all of the people with the knowledge and resources to move forward quickly), with no dissenters, (as a single dissenter could move things forward), to a common goal is not easy. It's effectively impossible.
If one of the many close calls had gone a little differently and a city were nuked accidentally, it would cause a global outcry and there would be yet another international effort to reduce the global nuclear stockpile to zero. And there’s a very good chance it would succeed. At the very start there was actually an agreement to not initiate a nuclear arms race but the Russians went back on it. So, for something that is “impossible”, we seem to see signs of it all the time.
The reason you think it’s impossible is because most things aren’t like this. Most things benefit some people and harm other people. This harms all people. Be a part of the solution instead of dismissing real solutions when you have no logical reason for doing so.
That's up for debate. I personally think AGI will be good, though not with a level of certainty that would allow me to use it as an axiom in a conversation.
> The reason you think it’s impossible is because most things aren’t like this
The reason I think it's impossible is because there are several other things like this and we did the same thing. When there is massive benefit to dissenters, getting everyone on board fails. Nukes, fossil fuels, deforestation, industrial chemical controls, overfishing, etc. are all examples of how we continue to fail at the exact same task.
Even Sam Altman has said similar things recently that he believes we're probably going to have to slow down and think about what we're doing.
I think a more advanced civilization would slow down and continue AI research on another planet (Mars?)...not here on Earth.
They achieve AGI first, that's bad. Communist AGI is a greater existential risk than us creating AGI. We must create it first so we can use it to harden our systems.
HN is scoffing because the timing of this petition is suspicious. To borrow your metaphor of “step on the gas”, this is as if the race had already started, several competitors are out ahead of the rest of the field, and now someone is yelling “step on the brakes!”.
[0]: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
As it stands, it says more about the author's egoistic view of humanity than of AI. Imagine for a second taking Feuerbach's The Essence of Religion and applying it to Descartes' rationality and Plato's nous. We'd get a critique of the rational human's construction of his own ideal - the essence of intellect.
AI threatens this ideal and ego-threats get sublimated into existential threats by those unable to understand themselves well enough to express themselves directly.
People write page after page about how it might kill us all in a flash, without ever offering a good explanation of why it would want to. My experience with people whose immediate reaction to something they're scared of is to destroy it is that they're panicky screechers who are an annoying distraction to the person actually handling the situation, whether that's wild animals, fire, or interpersonal violence.
I'm not saying 'just let me handle it bro, I totally got this.' There's a lot of potential risks, I don't think anyone is qualified to say they can mitigate all those, or even most of them. But I trust a machine intelligence - even one that's vast, cool, and unsympathetic - far more than the would-be Butlerians.
Two, I would generally not set aside an hour of my time for a video without at least some indication of what it's about. I'd rather spend that time reading than in the 'hot' medium of video.
Three, I find this video deeply suspect. It purports to document a 'private' gathering, yet it's clearly a well produced event that was intended to be documented and shared on video. People who actually want to keep a thing private are generally well able to keep it private. So while the participants have a valid point of view with many legitimate arguments, the facts of its existence suggest to me that it was absolutely intended to become public, and the 'private gathering!!' is essentially a marketing hook.
The point of much of the alignment debate is that people like Yudlowsky are pointing out that it doesn't need to want to, it just needs to not not want to enough.
You're hoping for an outcome ranging from "When The Yoghurt Took Over" to "The Metamorphosis of Prime Intellect", but many other people are expecting an outcome more similar to gwern's "Clippy".
Not saying there aren't good points. Only pointing out that many, many people can all view the consequences of this dynamic differently.
I am replying to the parent comment, but I too get that this petition is being signed onto by people at openAI specifically because they already made it with GPT-4. They definitely would not sign on to it if their (arbitrary) threshold for what to stop doing AI research with was a tool at a GPT-3 or GPT-3.5 level or higher or it included GPT-4 in their threshold, and not merely beyond it.
Hardly anyone in the tech community is capable of truly empathizing with the vast majority of people, who are born with none of the advantages that they had (genetic, parental, cultural, geographic, etc).
This is a gross exaggeration.
The same people SV people then turn around and say that these systems will help them get access to free medical care?
lol
Americans (let alone people elsewhere in the world) are already struggling. Recent reporting suggests a great many have to work multiple jobs. Almost all of us work an absurd number of hours per week. Many if not most can't afford homes. Plenty are ending up on the streets. Healthcare can literally bankrupt people. A vacation out of the country is an impossible luxury for most. The majority of Americans still don't send their children to college, usually because of affordability.
And I haven't even touched on what life is like in most of Africa or Asia.
This is the world we're bringing AI into. You have to be something adjacent to a sociopath to be okay with that. So long as our system is predicated on capitalism, AI may very well induce more downstream suffering than anything else humans have ever conceived.
If "AI" ends up making governments think they can continue to micromanage and get away with it, yes, that will cause more suffering. But that's not the failure mode any critics of "AI" are trumpeting about.
I agree. And I agree with your overall sentiment about the risks of pursuing AGI. I'm as cynical as anyone about the likelihood that the average person will really be any happier in a world with AGI (controlled by tech billionaires no less).
That being said, to claim that hardly anyone in the tech community is capable of empathizing with the average person is a wild overstatement that brings nothing to the discussion. Just adds to the noise.
Let’s get real concrete about what’s going to happen: people will lose their jobs, then their homes, they’ll become destitute, they’ll experience divorces, some will commit suicide, they will suffer desperately in myriad other ways due to economic disenfranchisement, kids will be deprived of a comfortable upbringing, etc.
How many in the tech industry are genuinely discussing the very real consequences of nonlinear degrees of automation for the kinds of ordinary people they barely interact with? How many are pretending that there isn’t something disgustingly immoral about having some of the most affluent and economically insulated people devise and inflict this reality upon countless millions?
I will maintain that this industry is morally bankrupt and nearly entirely devoid of empathy. These are not the people who should be in charge of our future.
Since the tone of your characterization is so absolute, why doesn't it apply to you? Why are you here in this tech community at all if the whole industry is so morally bankrupt? Why would "present company" ever be excluded for this or that reason? Because they're your friends? You're just projecting your own anger onto an entire group of people that you mostly don't know.
What I think you mean when you say all of this is that those in control of the tech industry are morally bankrupt. And, after 10+ years of getting kicked around as an engineer, I think I would have to agree. But I'm not so foolish as to broadly dismiss everyone in the industry just like me, who started out as a silly nerd who just liked computers and math, and who is essentially still that same person at their core, as a lost cause. I don't do that because I know everyone is fighting their own fight. But it's clear that those who aren't fighting are the ones on top that are sucking the life blood out of society. I'm more and more resentful towards that demographic every year. And I agree with you that they're crossing some kind of moral line by developing this tech, or at least by trying so hard to maintain control over it.
But the tech will get developed either way. If you're in the camp that thinks we should somehow just stop doing all this, you don't seem much different to me from someone that wants to mandate encryption backdoors. Our society will never be well coordinated enough to do that correctly. This isn't like making nuclear bombs, which takes a lot of physical industry. This is something that is just months away from running on commodity gaming hardware. Probably just a few years away from running on the average laptop. It does feel a bit like a harsh reality, just like the fact that a meteor could slam into the earth at any moment. But there it is; what are you going to do about it that isn't either futile or self-destructive?
A lot of people in the lower income brackets do the kind of work that an AI can't do. The people who should be worried most are actually college graduates doing clerical work, whose main work output is writing or evaluating texts. Even those people will likely use AI as a tool to enhance their productivity, because the AIs still are not good enough to replace people for tricky edge cases. The first companies that try to replace their customer support workers with an AI are going to have a bad time (and so are their customers!).
When almost everything can be automated, the problems that remain are the really hard ones that can only be solved by human experts.
A construction worker with a circular saw can cut boards way faster than someone with a handsaw -- but the introduction of circular saws didn't result in a bunch of carpenters getting laid off. Instead it made them more productive, and for people who get paid by the task rather than by the hour that is a huge benefit. They could build more and make more money, and a bunch of other people benefitted from their increased output, like homebuyers and property developers.
Similarly, as a software engineer I benefit from code generation tooling already. If that gets smarter and faster, I will be more productive, my team will be able to build software faster, and instead of laying people off I will expect to be given more work. Maybe our 4-year roadmap will be achievable in 1 or 2 years with the same size team.
Why so sure the end users aren't going to be feeding their own requirements directly to a Jenkins/Copilot/ChatGPT mashup running as a service in the cloud?
If you think this is the world "AI" is leading us toward, I think you need to think again. There are a huge range of skills that only humans can provide, and that will continue to be true. GPT-3 can't fix your plumbing or your HVAC when it breaks. It can't fix your car. It can't build houses. It can't build roads. It can't build computers. And on and on and on...
Nor are AI enthusiasts trying to get "AI" to do any of these things.
Humans are real-world creatures with real-world needs that we supply to each other by free markets, specialization, and trade. The "tech community" might take all this for granted, but that doesn't mean it just happens by magic. It happens because real humans do real work. "AI" is not going to take that over any time soon.
I've seen this abjectly atrocious argument about manual labor made by a number of people on HN and it's astounding how poorly thought through it is.
How? How will it do this?
Basically, people seem to assume that "AI" has some kind of magical power to do whatever bad thing they can imagine, and then they extrapolate from there. I don't see it.
> physical labor
If you think the jobs I described are purely "physical labor", or that the physical labor involved is so highly taxing that people can only do those jobs for a short time before they wear out, you definitely need to think again.
(You also definitely need to think again if you think those jobs are low paying. Plenty of people make quite a healthy living doing them.)
Will they still make a healthy living when there's an influx of laborers fleeing more automated parts of the economy?
GS just put out a report today estimating the impact of AI: https://www.businessinsider.com/generative-ai-chatpgt-300-mi....
I wonder sometimes if these accounts on HN making insane arguments that generative AI somehow won't be economically calamitous are bots. In fact, if I was at OpenAI and the goal was to avert scrutiny long enough to get to AGI, unleashing a torrent of AI shill bots might be near the top of the agenda.
Will those laborers have the skills required for those jobs?
> GS just put out a report
LOL--Goldman Sachs as an authoritative source on the impact of AI.
> I wonder sometimes if these accounts on HN making insane arguments that generative AI somehow won't be economically calamitous are bots.
You must be joking: you actually have trouble telling posts by bots from posts by humans? Even with a large number of samples? (Never mind that you can also look at the account's profile page, which will give you very useful information.)
True today. What happens when the other industries collapse and there’s a flood of labor into these industries? Sure - initially the experienced and skilled labor will continue to command a higher price - but over time supply of talent will drive that down too.
We're working on getting AI to do all of those things, yes.
>GPT-3 can't fix your plumbing
GPT-3 might not. But GPT-4 fixed plumbing in a test done by Microsoft Research before it was even publicly released! It's in the "sparks of AGI" paper.
GPT-5 is almost done training now and other language models are already embodied and moving about in the physical space. Your objections are behind the times, and increasingly so.
Near the end:
> If you don’t feel comfortable doing this, you may need to call a plumber.
---
Source: https://arxiv.org/pdf/2303.12712.pdf
Sparks of Artificial General Intelligence: Early experiments with GPT-4
[page 53]
---
The computer is trying to solve a problem. It can tell the human to do anything, one thing at a time, and one line at a time.
PROBLEM: there is water dripping from the kitchen ceiling, and you have to figure out why and fix it ----<|endofprompt|>
Human: The kitchen ceiling is dripping water
Computer: Check to see if there is a bathroom or other water source directly above the kitchen.
Human: Yes, there is a bathroom actually
Computer: Check the bathroom for any signs of water leakage, such as puddles on the floor, water drip ping from the ceiling, or water stains on the walls.
Human: There is a little water on the floor
Computer: Try to identify where the water is coming from. Check the toilet, sink, and bathtub/shower for any leaks.
Human: It is next to the bathtub and the toilet
Computer: Check the seals around the bathtub and toilet to see if they are worn or damaged. If they are, replace them. If they are not, continue investigating other potential sources of the leak.
Human: I can’t see the seal around the toilet, there is a porcelain thing around it. I don’t see any damage in the bathtub seal though
Computer: If you can’t see the seal around the toilet, you may need to remove the porcelain cover to inspect it. If you don’t feel comfortable doing this, you may need to call a plumber.
Human: Ok, I removed it. But I can’t tell if it’s damaged or not
Computer: If you can’t tell, it may be best to replace the seal anyway, just to be sure. If the leak persists after replacing the seal, you will need to continue investigating other potential sources of the leak.
Specialization is useful up to a point. But I like my general intelligence, and rather resent its devaluation under your beloved 'free' markets.