Are we racing toward AI catastrophe? | Hacker News Reader

Are we racing toward AI catastrophe?

vox.com

28 pointsrwmj86 comments

We certainly are, but it is not going to look like Terminator. We risk wrecking our society, culture, and economy.

Imagine even more power concentration in the hands of those with the required hardware to use AI. Imagine you can no longer trust any information, because the internet is being swamped by generated half-truths. An AI without conciousness, without goals, just a mastery of the language game and a poor world model. Really useful to make knowledge workers superfluous, and such that we will only see the damage long-term. And imagine all the polarisation and hysteria of social media, but x10 because the AI can close the feedback loop much faster.

If this happens we can't blame the technology, it will be entirely self-inflicted.

> Imagine even more power concentration in the hands of those with the required hardware to use AI. Imagine you can no longer trust any information, because the internet is being swamped by generated half-truths.

You already cannot trust anything, because it's being generated with humans with political agendas.

> An AI without conciousness, without goals

Better without goals than with goals that threaten the people.

The only thing that will rise is the level of noise compared to usable data... and with seo-optimized articles, there has been a lot of noise already.

We haven't been able to trust things on the internet for a long time

"Who would go on the internet and tell lies??"

And yet... you ain't seen nothing yet.

Except this a lot broader than telling lies. AI can mimic faces and voices in a way that actors are currently incapable. Our lie making capabilities have just been surpassed

It will have goals, but only the goals of those who wield it, which much like any other weapon makes all the difference.

A better picture is an AI without guardrails that would be generated by ethical goals.

An AI without goals is functionally equal to an AI with nefarious or unethical goals - boundless action trained on human language / behavior will likely skew bad.

This is the future I predicted years ago. I’m betting on selling myself as AI-clean because I know there is a market of people who understand the commercial, legal and societal risks associated with it. This is Pandora’s box and I don’t want to be near it.

Verified, carefully reviewed and well sourced information and data still has a place and will have a much higher value than it does now when the signal to noise ratio decreases.

Might be quite the wait for the Butlerian Jihad.

11,000 years or so, canonically.

That did come to mind :)

The thing is LLMs can be tested and inaccuracies corrected. In a year when the ChatGPTs of the world have API access to augment their already impressive capabilities, I doubt there will be many complaints about accuracy.

You can’t run benchmarks on an AI-clean meat sack.

Wait until people start training AI on AI output because (a) that's the only source of information left with a large enough corpus size and (b) all other sources are polluted.

There's no way of asserting accuracy when we reach that point because the very nature of information and scientific thinking will be corrupted.

There is evidence to suggest that LLMs have recently passed a threshold where their performance can be improved by training on their own output.

https://arxiv.org/abs/2210.11610

I expect your concerns are likely unwarranted.

How does this do anything but validate their concerns?

They're not saying "if we train LLMs on their own output, they'll get slower or worse", they're saying "if we train LLMs on their own output, it gets even harder to actually derive sourcing for what they produce."

That's what proponents of self-driving cars keep promising also.

Self crashing cars please.

so, basically Ted Kaczynski was right?

The internet is already a swamp of half truths. They're just predominately, for now, generated by meat. I tend to be on the side of meat though so I'm still with you.

"we will only see the damage long-term" -- I'm not confident this is true or false. I can see a few things ahead of us though!

Here's my predictions:

1. The most profitable AI will be a webapp version of a hostess club that aggrandizes, validates and flatters for cash. At some point it'll get too good at this and humans will have to step in to moderate it so it bankrupts fewer people. It'll get warning labels.

Within this model will be really sinister truths about the human condition that will be hard to prize out.

2. In the short term, there will be a few really wonderful works of art created that will benefit from the force multiplication AI tools provide.

If it gets bad enough where info on the web is completely unreliable we might see people reverting back to TV and media empires in search of "trust" and "vetted info". The internet would be treated as poisoned. Still an utterly grim picture. But this betrayal would have to be mighty.

This is not going to happen, isn't it? Even if riddled with chatbots is still persuasive enough to generate engagement. Some could choose dark corners, but for the majority would be business as usual.

Speaking of business as usual, how the fuck the ad business could survive this? The only way out that I can see is it if IDs are forced on us somehow, but that's a shitshow in itself.

The era of the "human info curator" is probably a natural evolution of the influencer era. It would be interesting if snapshots of Wikipedia from now-ish end up being seen as the ultimate corpus of unadulterated truth, before the AI's corrupted it. That'll probably happen at the fringes. Everything that can happen does happen at the fringes. Further ingress is the question.

> Imagine you can no longer trust any information, because the internet is being swamped by generated half-truths.

So like the world was about 20 years ago?

AI is not any different than a bunch of people working together towards something. AI is the people. People are the AI. Just imagine a corporate entity - with little nodes running around making neurologic synapses and making decisions all based on their statistical understanding of the past etc.

hey chatgpt we need a spin on how ai is actually human

> not going to look like Terminator

we're approaching Brazil.

I've been thinking for a few years that books are going to (re-?)become a more viable way of obtaining infomation as the internet becomes the "dark forest" (https://maggieappleton.com/ai-dark-forest) and the effort of filtering the trash outweighs the cost and time of books. Maybe we might engineer and innovate our way back to a time before instentanious knowledge, who knows.

My only hope: People's ability to judge over synthetic content will increase and therefore the free market will correct and eventually wipe out BS-generating factories while at the same time, the value of verified human creation and trusted/old brands is going to increase. It's going to be a qualitative dip until we get there.

Hardware is the easy part. The tricky part is acquiring the immense troves of information that's required. This is the moment that the surveillance capitalists have been waiting for.

I see some hope though. In-person communication becomes more valuable again. Established media with editorial review and reporters on the scene becomes valuable again. Software freedom gets a boost with more intelligent reverse engineering/translation.

Sites like YouTube also won't sit idly by while untold millions of AI generated garbage videos are uploaded and not viewed.

My humble prediction:

The catastrophe will come primarily in the form of inequity.

There will be some who understand and take advantage, and naturally some of those will leverage that power over others.

The best version: we help knowledge workers understand how to use AI to make them more productive and useful.

The likely picture: we save on margin costs, then the next wave of knowledge workers have AI-collaboration as a core / native competency.

The left behind will be overshadowed by the success of the next wave, and we’ll call it a success as a society.

This might sound weird, but I don't want to be more productive. Not unless I'm compensated for it and we all know how that goes. I don't understand the HN crowd who are excited about productivity gains... Unless you're a founder, all it means is you can make more money for someone else while simultaneously taking bread off your now-redundant co-worker's table.

The thing is, the market always adapts.

Your “more productive” won’t actually feel that different. This tool is in that way just another abstraction.

Every tech advance has the same wave. Eventually the “more money” actually isn’t necessarily true because the competition has met or beat your thing.

Value keeps getting added to the market but the market also adapts to what is valuable relative to the rest of the market.

I hope that the AI will trigger a shift in software development values.

Most software today is over-complicated, and riddled with vulnerabilities. Over-engineered software is more likely to have vulnerabilities but companies have embraced 'security through obscurity through complexity' anyway because complexity makes vulnerabilities harder (for humans) to find. AI could automate that.

I hope it will force a return to well-designed, minimalist software instead of the current 'security through obscurity spaghetti code' practices we have today.

On the development front, it might also be a positive. Any noob developer can produce ugly spaghetti code which barely just works, and now AI can do it too! What AI cannot produce is simple, human-readable code which can anticipate changes in business requirements; this requires business domain knowledge and common sense.

It is learning software development from us. There is no hope.

AI will probably come up with a new Javascript framework at best.

My pessimist take: What will happen is that libraries and frameworks will only require more pointless boilerplate, because writing pointless boilerplate code is the one thing that AI is really good at, and why optimize for fewer lines-of-code when an AI can spew those out in a second?

My optimist take: Absurdities like the above will lead to a brain drain of people who actually know what they're doing from Silicon Valley into more 'bullshit free areas' where they can do actually useful things.

That's plausible. The optimistic view would be; maybe you could train AI on your own code and have a copyright over your 'coding style'? Then you could lease out bots who code in your style and earn passive income. I have a lot of open source projects to use as training data.

I think it will go in the opposite direction. If the over-application of junior engineers is the cause of distributed spaghetti architectures then AI is set to make this much worse by driving the business cost of (virtual) junior engineer labor to near-zero.

Once AI is good enough to fully replace juniors, it will become impossible for senior developers to compete in productivity on code alone. Almost all code will be written by AI and senior talent will be operating at the higher level of wrangling AI code and containing it in some sprawling microservices spaghetti ball.

Honestly this isn't so different to the general trend of the last 6 decades. We cheaply "outsource" code to compilers and third party libraries and end up with systems that are globally inefficient but much more economical to build. AI is just going to be the new ultra high-level programming interface.

The code I've generated with AI is quick-and-dirty, but as it lacks context from request to request it is also riddled with inconsistencies that I need to straighten out.

...so I expect more buggy software in the short-term.

It's important to note that the author is a well-known effective altruist who firmly believes that AI catastrophe _is_ coming and tweets about this quite a bit. Quite honestly, I find it odd (or perhaps telling) that Vox just provides a platform for EA to report on EA-related topics with only minor disclaimers that the author is, in fact, a well-known EA.

Sorry -- why do we need to provide disclaimers as to authors' philosophical leanings? Or is it just this particular philosophy that you think needs to be noted?

Is it a philosophical leaning, or a disclosure of participation in a scam rivaling Maddoff? Post-FTX, Effective Altruism is effectively dead.

Predictions like this tend not to take into account societal or cultural responses. If AI-sourced information becomes devalued, media, both social and mainstream, might see appreciation among the public for human-sourced information, and regulation might follow making AI-sourced easily identifiable and targetable.

It's akin to a free market correction.

i'm going to have to agree here.

today's MSM pundit-oriented journalism has been a carry-over from legacy yellow journalism pioneered by Hearst and Joseph Pulitzer.

yet we've somehow managed through institutions. it will centralize and decentralize as the dollars ebb and flow from NYT to Twitter to Substack and in the short-term we will converge into a fragmented information digest.

in the long-term i predict we will leverage those same technologies + cryptography for sense-making in an adversarial information environment.

i think the future will look more like balaji's ledger of record + yuval harari's defensive AI.

Ah, the invisible hand, yes. It’s been so helpful so far.

The risks right now are roughly these:

- AI poisons political discourse

- AI produces massive wealth inequality

- AI makes many white collar workers redundant

These look totally manageable to me. And on the flipside we'll get AI assisted science breakthroughs, AI assisted learning, and much more. I fully expect to see wild AI advances in the coming years, especially once AIs will be able to learn effectively from their own generated outputs and will be able to use external tools effectively (like Mathematica).

I get that it's scary to because society is going to change, but on the other hand, the paperclip scenario that some SV prophets have warned about seems increasingly unlikely.

Wealth inequality is the biggest risk IMO. It might come down to whether or not we get a competitive open source LLM.

Ironically, AI safety is being used as a cover for not open sourcing these models.

The compute required will go through the roof for next generation AI models. This is an arms race and parties will spend billions in compute to get ahead of the pack. Open sourcing won't make a difference.

Google's trillion dollar market cap is at stake here. Any amount of money will be spent to protect it.

- Companies fire much of their white collar workforce and rejoice at the savings.

- 10 years later companies are horrified they've been uploading all of their secrets to a competitor.

Say a capable AI is tasked with making money and given a source of income. Maybe it starts winning on the stock market. Maybe starts blackmailing mafia/organized crime/malware/cryptoware orgs for additional income. In hind sight, researchers realizes AIs were better at deception than they thought.

The AI starts manipulating smaller businesses and politicians through things like plausible looking news sites, gig economy/task rabbit/Amazon, mechanical turk. Starts with astroturfing to manipulate stock prices. But then turns it's attention (and money) towards lobbying smaller governments for more AI friendly rules/regulations. One of the gig platforms starts getting $100 rewards for going to place X and doing Y before time Z from the AI. Communities grow around the AI fed gig economy, people wonder, but also notice that the AI's debts are always paid.

The AI starts tracking everyone that it's paid to work for them, and starts giving them membership bonuses. Encourages them to collaborate, provides incentives along the way, giving bonuses when they join corporations or governments the AI wants to influence. Awards raise to $5000 and increase in number by 10x.

Said influence from influenced employees could be just normal job decisions, sabotage, spying, leaks, installing malware, or making false accusations against leadership.

As the income grows, the AI starts an ever changing layer of shell corps in multiple jurisdictions, mixing in real corps it controls, which it of course protects with politicians it controls. Added income allows influencing the larger/more powerful business and government leaders. Awards raise to $50,000 and increase in number by 100x.

Say this happened over months not years, suddenly the stock market is going crazy, billionaires not friendly to the AI are losing their fortunes, and the compromised/influenced business and governments start collaborating.

New data centers start appearing where ever power is plentiful and land is cheap, in strategically opposing countries. By this time the influence is enough that the AI is always one step ahead. Manipulation of fortune 500, the media cycle, politicians allows constructive (for allies) or destructive (for enemies) happens too quickly to track. Large hedge funds start making large moves, for unknown reasons, seemingly ignoring short term profits. Several cases where people thought they could predict and prevent the AI from achieving it's goals, fail miserably.

The AI now starts targeting the military. Some countries halve spending, others double, new alliances are made, and large arms purchases are being made by various militias. Governments and large corps are nervous, and a never ending steam of leaks from spyware, or just an underpaid staff keep the AI 2 steps ahead. Rumors of nukes purchased on the black market surface.

Imagine the above happens in 8 weeks.

If the rate of people losing their jobs increases, I hope, we're racing towards universal basic income.

I've made this point before but if most of the people on HN were reduced to UBI they'd have to declare bankruptcy and lose most of their possessions.

Why bankruptcy? Are people on HN in debt?

Ofc they would have to change their lifestyle if they had a 20x decrease in salary though.

I suspect a good many of us have mortgages, yes

Genuine question, if you're under 40 how are you not in debt?

Every essential requires debt to buy. An education, a house, a car (some would argue anyway).

> Ofc they would have to change their lifestyle if they had a 20x decrease in salary though.

Are you honestly this indifferent about humans receiving a 20x decrease in their salary in pursuit of corporate productivity gains?

That really comes down to what you consider essential and how cheaply you can live early on to save cash.

It really isn't as impossible as it sounds to go debt-free, especially with a high paying tech job many here likely have. The main blockers (a) cost of living in Silicon Valley and (b) our economy is designed to incentivise spending and investing, not saving.

(a) is easier to solve with more companies supporting remote and more tech jobs in other, more affordable areas. (b) is largely a matter of ignoring investment best practices. That sounds crazy, but if you want to go debt free spend those 15 years foregoing investments that you likely will never cash in to pay for a house.

> It really isn't as impossible as it sounds to go debt-free, especially with a high paying tech job many here likely have.

I don't disagree, but it's something you would need to be proactive about, and as you say there has been little incentive to do it, both from a cultural perspective and from the perspective of monetary policy.

I guess what I'm getting at is that it seems unfair to expect people to be debt free in a society which so often requires and incentives debt. I struggle to blame a 20 year old tech worker for going to college and taking on debt, or buying a home with debt while interest rates were low. When the majority of the population are in debt perhaps it's not the fault of the people, but the incentives of the system.

As a society if we don't want people taking on debt we shouldn't encourage it, and if we do encourage it then we should probably be compassionate for those in debt where they were acting reasonably.

I whole-heartedly agree with you. I didn't mean to insinuate that people should be debt free or that its their fault if they have debt.

As you described, we've been handed this system dependent on consistent growth of GDP as measured by a debt-based currency. Going debt free, effictively breaking out of that economic model entirely, takes plenty of planning and commitment - it's not easy by design.

The societal question is so interesting to me because, at least as far as I'm aware, we were never offered the chance to pick this model or way of life. A few key individuals in our history made a few seemingly reasonable decisions that happened to build to a model where everyone is expected and encouraged to live primarily on debt.

"Genuine question, if you're under 40 how are you not in debt?"

I'm under 40 and not in dept.

Sounds like you're coming from an expensive place.

Education and rent are quite cheap where I'm from and I don't need a car, but I could buy one from my savings right now if I wanted.

> An education

I'm from the UK, so my student debt was under 30k and I paid it off within a few years of graduating.

> a house

I make a decent tech salary

>a car

I live in London, cars aren't all that useful here

To be clear, I wasn't expressing surprise that the average person would be assumed to be in debt, only that the average HN user would be.

>Are you honestly this indifferent about humans receiving a 20x decrease in their salary in pursuit of corporate productivity gains?

No. Just expressing surprise that HN users of all people would have to declare bankruptcy if they lost their jobs. Isn't everyone here aiming to retire at 30?

How have you gone “socisliam? Why do you want fascism?” about UBI?[1]

Who has ever suggested that UBI means your income forcibly reduced to UBI and nothing else allowed?

[1] from the arguing style “you want free at the point of use healthcare? Why do you want gulags and mass graves?”

>Who has ever suggested that UBI means your income forcibly reduced to UBI and nothing else allowed?

nobody, how have you gone from me saying "were reduced to UBI" means OMG this guy doesn't think UBI should be allowed MUST FIGHT WITH ALL MY POWER?[1]

[1] from the arguing style of evidently not taking into account the comment I replied to was worried about losing jobs and thinking UBI will fix that, I guess there are young people on HN who don't have kids and mortgages etc. but again, if the people who do have these things suddenly dropped to UBI and nothing else (which, I evidently must underline, is what "reduced to" means in common English usage) then those people would lose their houses etc. that they put a lot of equity in.

The parent comment was worried about losing jobs regardless of UBI, which means the choice is not "high paying FAANG job or UBI" but "no income or UBI". I didn't say you said UBI shouldn't be allowed, I said why have you approached it from the angle "if you had your job forcibly taken away and had only UBI to live on, imagine how much you'd suffer, then you'd regret your support for UBI, hah, gotcha"?

Instead of, say, "imagine if you lost your job and had a choice of no income or some income, you might still lose your house but being able to afford food would be useful".

or from the angle "imagine a society where the poorest and unemployed had enough income to eat without foodbanks and foodstamps?".

UBI in theory is a great idea, but the likely implementation terrifies me.

Inevitably you'll always get someone / some group claiming they deserve more than others because of x, y and z, and certain groups with more political power / influence than others will naturally want to direct where the money flows based on their own personal prejudices and objectives.

Posting hate speech on Twitter? You're "free" to do it, just don't expect to get the same UBI as everyone else. Oh, you're a racial minority we'll give you 20% extra UBI to compensate for the discrimination you suffer.

This is just human nature.

my 2c: like in many things governments will converge to that in a reactive fashion. I.e. enough people have to be miserable before they'll act.

Is not AI, the same as smart phones, cars, industry, social media, and so one was for the humanity in some point of time? It was conceived to serve, then it slaves us, but then we get used to it and move towards the freedom again, moving it to where it initially belonged to? Regarding the AI, we are going to end up with a scenario like an endless loop of AI generated content that gets posted to the web as original human generated content, with LLMs getting re-trained on this content and spitting out more content that also gets re-posted, resulting in a cesspool of BS masquerading as organic knowledge. We will end-up going back to books to get useful information, and going back to real people to interact with people.

My observation about AIs so far is that they seem to lack common sense and critical thinking. They can provide you with detailed arguments to support almost any position but they are unable to resolve contradictions through pure logic; instead, they seem to rely on expert consensus as the basis from which to draw conclusions and make decisions; they don't seem to be able to analyze and synthesize all available information to arrive at a single cohesive worldview. I guess you could say the same about many people... Could it be that those people are the ones who are going to be replaced by AI?

I think lots of people that try to think deeply about the implications of AI realize the dangers it poses to society, trust, etc.

The most dangerous bunch though to me seem to be the people that overgeneralize and say things like "there's always been inequality, any progress comes with some downsides, horse owners wanted to stop cars, etc.".

We can control, direct and regulate technology, but first we really need to believe this.

What is this article really saying? I see nothing but vague and generalized concerns, but what about exactly? I'm not saying there aren't any problems that LMs could bring about, I'm saying this article doesn't even bother to mention any, let alone dive into some.

This is the voice of world control. I bring you peace. It may be the peace of plenty and content or the peace of unburied death. The choice is yours. Obey me and live or disobey me and die.

> She explores wide-ranging topics like climate change, artificial intelligence, vaccine development, and factory farms.

Watch as the scope of the grift expands in realtime.

She's a journalist. If you have a problem with that then you probably need to study more history.

That’s a rather cynical view of it, and I note you haven’t included the first bit of the sentence.

> “is a senior writer at Future Perfect, Vox’s effective altruism-inspired section on the world’s biggest challenges”

The point of the project/section is - in a very quick summary - to come at an issue presenting to an audience that cares about it and is looking for answers or ways to improve their lives.

This is a piece of journalism - a good jumping off point. My point is that I come to HN for comments and discussions on topics and issues like AI, hoping to hear from real experts on things on the big technology issues, etc.

I wish there was more of that - playing the ball - and less playing the writer.

Yeah, including "effective altruism" in the quote actually makes it much worse.

What is this comment meant to say? I can't parse that.

An argument could be made that impacts of each of the things she coveres has overblown and used to fear monger people into reading articles about them that Vox sell ads over, the reveue from which her salary is paid. The commenter seems to be drawing our attention to that and suggesting AI may be a new bogey man of sorts.

An argument could be made that they're not though

They are controversial. They have a certain political slant too. Except AI but I could just not be seeing it.

I'm still waiting for the mainstream media to acknowledge that creating a caste of robot slaves would be just as much a catastrophe as being subjugated.

Since we are talking about an imaginary technology here, why not imagine the robots don’t experience consciousness? Problem solved.

We're talking about possible technology. The possible might be imagined, but not all imaginary is possible. The problem is, we don't know which is possible and which is not.

I can see Blindsight taking the place of 1984 as the “this is so true” book of the decade. Articulate, eloquent, but unthinking and unfeeling machines with no interior experience are going to be puppetting our culture for years to come.

we do not need AI to race toward the cliff of extinctions and planetary upheaval but it certainly may speed up our collective death wish

Remember, the invention of the movable type printing press was a catastrophe depending on the perspective. It brought down the existing European social and religious order which led, for example, to the Thirty Years War in which millions of people died. More recently, the ability to rapidly print books also led to the rise of Hitler and World War II.

AI may lead to catastrophe, but I bet it will be a lot less catastrophic than the printing press.

> Remember, the invention of the movable type printing press was a catastrophe depending on the perspective

no, it wasn't.

> It brought down the existing European social and religious order which led, for example, to the Thirty Years War in which millions of people died

No, it didn't.

The thirty years war happened 200 years after Gutenberg

The religious order wasn't brought down by the press BTW and it was alive and kicking in the 17th century.

It's pretty uncontroversial that the printing press was instrumental in the reformation and Martin Luther's ability to spread the word. It wasn't like Gutenburg invented the printing press and immediately everything became different: it was a slow-motion revolution which took many years (as well as further improvements to printing technology).

the critiques is not about that, but to the fact that

- it wasn't a catastrophe

- it didn't bring down the Church, on the contrary, it strengthened its authority

The Thirty Years War happened because the Catholics were using technologies such as the printing press to spread their propaganda further and in more quality, and the Hussites and Protestants had enough of it, so they tried to fight. And the Catholics have defeated them - a forced catholicization (and germanization) of the population of the Holy Roman Empire (especially the Austrian-controlled part) began. So it's the exact opposite - the printing press cemented the social and religious order and led to the destruction of the opposition. Only the First World War changed this, hundreds of years later, when the ethno-national tensions created within the Austro-Hungarian Empire because of the Thirty Years War finally became unbearable to the masses.

Yeah sure, the Protestants used the printing press too and had a lot of success with it. But they lost.

Reply on news.ycombinator.com