We could stumble into AI catastrophe
cold-takes.com
cold-takes.com
Things to worry about in the near term:
- Really effective targeted advertising. Most computers already have a camera watching you. That's now coming to TV sets. Amazon and Google are always listening. So far, all this info is only used to select ads. Soon, it should be possible to generate customized marketing content for each consumer, and use immediate feedback on how the customer reacts to adjust the sales pitch. Alexa is already part of the way there.
- Machines should think, people should work. That's what working in an Amazon warehouse is like today. The machines tell the humans what to do. They have to; only the computers have an overview of the process. (Yeah, Marshall Brain's "Manna", which everyone here has probably read.)
- Big Brother is watching you. It is possible now to watch most of the people most of the time. Track who's out of view and for how long, to know what's being missed. China leads in this, but the UK is not far behind. The US isn't centralized enough to integrate all the available data yet. We're coming up on Oppression 2.0. Latest advance - Iran is using surveillance cameras and face recognition to catch women not wearing hijabs.
- Machines beating humans at business. In some areas, AI systems may generate better returns than humans. That already happens in parts of finance. After all, it's an optimization problem. The free market may force companies to use AI more in management.
- Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase.
This is all next 5 to 10 years stuff.
you seem to implicitly assume that a college education is an unnecessary expense for most and that therefore, it shouldn't be really given away so readily. this suggests to me that you assume that the real purpose of college is job preparation. your reasoning makes sense from the perspective of a higher level institution (or corporation, or possibly a government) seeking to be as efficient as possible regardless of the impact on typical human individual's well-being.
college is not a 'factory' (or any sort of industry) that 'manufactures' workers for companies.
The only reason people are willing to pay tens of thousands of dollars for college is that it is a credentialed degree that people think will open up career doors. We shouldn't beat around the bush on this fact.
I agree, but you won't have a sheet of paper proving it. It's like that infamous scene from Good Will Hunting. Question is whether you want an easy way to PROVE you're educated. Which is what a license, certificate or diploma grants you.
Why else would you need to prove your education except for work?
Essentially a new way to establish hierarchies.
But we're nowhere close to that. Also, if it's being used to establish hierarchies, I could still see that proof being important.
Getting “there” doesn’t happen overnight, but the point is that we are already heading there, and to the question “why get a degree if not for work”, I presented a potential future option.
> I could still see that proof being important.
This is key to the point I was trying to make. Essentially that the purpose of education may change over time, but the value of gaining an education will still be there in one form or another.
If learning is “merely” a diversion, I don’t see why people would think about it any differently. Whether or not it still makes sense for that proof to carry a six figure price tag is a separate conversation.
If you want to learn about chemistry or biology for learning's sake, good luck. Most autodidacts won't have space in their apartment for a -80C freezer or flammables cabinet, and their landlords might not approve the installation of a proper fume hood.
We have things very easy in software. The cost of iteration is low, you can fit a respectable laboratory in the space underneath your desk, and you're unlikely to accidentally poison or maim somebody if you screw up an unwise experiment. Not many fields have that luxury.
Yes. That changed on February 28, 1967.[1][2]
[1] https://www.chronicle.com/article/the-day-the-purpose-of-col...
The spike in demand drove a ton of "normal" schools (teachers' colleges) to become proper colleges and even universities, plus a bunch of new ones to pop up, but rather than increasing the supply of (previously) college-level education, it mostly created a kind of knock-off, lower-tier product—there just weren't enough excellent professors to meet demand, not enough good, experienced college administrators, not enough anything, including, arguably, students who were fit for college (as it had previously been) in the numbers that were now attending.
Unfortunately, the vocation focus of college has been a driving force since the early days of the industrial revolution, dating to the 1850s. The Morrill Land-Grant Act explicitly states that colleges (at least in the case of land grant schools) are there to train people for work:
>without excluding other scientific and classical studies and including military tactic, to teach such branches of learning as are related to agriculture and the mechanic arts, in such manner as the legislatures of the States may respectively prescribe, in order to promote the liberal and practical education of the industrial classes in the several pursuits and professions in life.
I respect and value the other traditional purpose of college, but we can also acknowledge that these purposes can change as society does.
In seriousness, I feel like the only hope here is that the general population gets more and more aware about the negative impacts on all this hyper-targeted advertising. Certainly we've been making strides there with things like GDPR and documentaries like the Social Dilemma, but with so much money on the line it will be hard to fight back.
> - Reduced need for education.
This subset is the radioactive bits of AI impact. We can always address the rest, but that requires thinking and an education.
Not only this, but the realtime trade-to-trade performance of companies is also easily available. If I was building an AI, I would let it make realtime decisions on where to put money as the "actuator" of society and advertising/consumer spending metrics as the sampling mechanism.
Whereas in the past there was some breathing room for society to catch up, evaluate and control this technology [0], today the rapidity has converted it into a wild west: tech entrepreneurs and their financiers declare their self-serving moral standards as the new norm, the broader public is dazed, confused and misinformed and public institutions, nominally at the service of the citizenry are only as good as their weakest, most capturable link
Name one development over the past decade that indicates we [1] are actually capable to control what is happening and the above dark picture is mere scaremongering.
[0] think e.g., about regulatory constraints around insurance or financial data and models
[1] as in citizens anywhere in the world
This is not a matter of “if”, it is a matter of “when”.
I've long suspected China had the right idea here, and am, in this case, not happy to be proven right.
Especially what do you think China's idea is?
And also how that idea relates to, in your words but abbreviated, «choosing between a network and democracy».
Thank you.
This was part of what was going on in the recent novel "Mountain in the Sea". An AI-controlled fishing corporation figured out that replacing robots with enslaved humans on it's AI controlled fishing fleet was more cost effective (the ocean air was hell on the robots apparently, they kept breaking). Each fishing boat was controlled by an AI which had the objective of maximizing profits. As a result the oceans were being depleted of fish at a rapid rate.
I actually run some Manna-esque systems in my own small business (most notably a system for stock picking and shipping) and this isn't something to worry about in the near-term.
It greatly, greatly reduces the amount of frustration and stress people experience in mundane jobs, greatly reduces cognitive complexity and just generally makes the work more satisfying.
Maybe there's a risk of humans losing their jobs to machines, a la chapter 2 of Manna, but to be frank it's no different to the threat of outsourcing which has existed for 50+ years.
This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both digital and physical, to give them as much tangible impact as possible.
With the rapid rise of AI capabilities, where groundbreaking advances are being measured in weeks rather than years, I strongly urge developers to think harder about AI safety. AI safety shouldn't be a dirty word that has quotes surrounding it to delegitimize it as a farfetched concern. Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.
We're just digitizing that.
We have that now. It's called a corporation. See yesterday's article about Exxon.
ai generates content <--money--> ai pushes ads to promote its content.
It's somewhat already happening: recommendation algorithms choose what content we consume, analytics make decisions from this data and make more content that recommendation algorithms like. AI HFT bots can directly influence economy and so on
Or rather, they're giving the rest of us a glimpse of what life has been like for those at the "bottom of the pile" for a long time: a world organised around goals essentially completely unrelated and indifferent to those people's existence.
That’s banal, self-interested human evil. Corporations and governments alike have perpetuated mass crimes and cover-ups.
[1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
I think there’s a good argument to say no, we don’t have that now, and these are some ways it could be very different.
Video: Why not just think of AGI as a corporation?
An AI could be completely untethered from that and set new thresholds for evil corporate behaviour.
An AI could be completely untethered from that and set new thresholds for ethical corporate behaviour that operates on fundamentally ethical foundations of caring for the environment and humanity
1: https://www.etsy.com/news/business-as-a-force-for-good-defin...
I wonder, with crypto crashing and potentially becoming worthless, if all those mining cards will get sold on cheaply and repurposed for AI and we see another explosion in AI capability.
My pet theory is that consciousness could be (not saying I know for sure this is the case but it could as well be) an effect of "survival instinct". That is: "i'll take action to not die".
Now this is very easy to learn from us as we do this all day long so to speak. Most of the way people react on the internet is tinted by the fear of death and survival for example.
What Im saying is that what we intuitively think as very complex could actually be very simple. As simple as some task optimization. Knowing the only thing that could unplug an AI is us, that'd lead to a hell of a fight. And maybe our complete termination.
This thought is entirely the result of evolution. Beings that didn't have a survival instinct didn't live to reproduce, so they don't have descendants that we can observe.
There's no guarantee that a survival instinct is a necessary or even desirable element of conscious thought. You're just describing survivorship bias in its most literal form. cue the picture of the shot up airplane
Ofcourse, in our foolishness, we could create an ai with the goal of survival and replication at the expense of everything else. Or self optimisation could inadvertedly do so
It's not 100% clear that consciousness is necessary even for things like creative problem solving or goal-setting (i.e. choosing which problems to solve). That is, advanced intelligences that can do all we can and more—and could even "decide", in the worst case, that they'd be better off without humans—may not need consciousness as a precondition for that.
Cf. Peter Watts' Blindsight
Imo we should have ChatGPT (and future iterations) in a self-sustaining underground bunker somewhere and an intuitive radio interface connected to it. If civilization collapses for one reason or another, all the next civilization would need to do is re-invent the radio to experience a rapid acceleration of technology.
Agree with me or suffer the basilisk.
Star Cops thought about it more than our popular journalists or philosophers seem to think we should:
-----
Michael: ...it was slightly sensitive material, and in my field one is aware of the vulnerability of carrier waves.
Nathan: Is that what you do here?
Michael: It wouldn't be wise to read too much into a casual remark, Commander.
Nathan: That's your principal research area here, isn't it? Communications monitoring.
David: Not more buggers and supersniffers...
Nathan: More impressive than that. We've always been able to hear more than we can listen to. Are you developing intelligent listening systems?
David: Sounds impressive.
Michael: I must suggest that to go on could be a foolish risk.
David: Threats, Doctor?
Michael: Observation, merely.
Nathan: Machines that listen to everything and decide for themselves what's worth passing on? Is that where your BHG information came from? Did you do a test running? Did one of your new computers pick it up from the babble of all the world?
Michael: Commander, it's a pleasure meeting you.
David: Well, tireless attention to every word spoken? No possibility of human error. You can tell it to listen to the bad guys, it'll listen to everyone and identifies the ones you [won't like.]
-----
The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.
To a degree. (See: piercing the veil.) Also, corporations need an authorised signer, who under current law must be a natural person.
Having algorithms and AI models decide how to profit regardless of how it harms humans is the inevitable next step.
A poetic sentiment, but does not hold in reality. Anyone who has spent more than ten minutes being a square peg in a round hole can surely recall their frustration when a bureaucrat is refusing to do the common-sense thing, because The Computer is telling them something that is complete and utter nonsense.
Once The Computer is part of any human process, its decisions, be they inscrutable, insane, or simply stupid are not to be questioned. The line drone that is using The Computer has no agency to countermand it, the line drone's manager has no agency in rebuilding, or re-configuring it, and the executive that authorized the system will never even be made aware of the problem.
These problems exist when The Computer is simply executing the rules the programmer fed it, and will persist in the same way when The Computer is executing an inscrutable neural network. Except it'll be worse, because neural networks are capable of executing even more tasks poorly.
-----
In theory you are responsible for, and control the heating element, but in practice, we're the society from Idiocracy that cooks everything by sticking it into the microwave, and pressing the 'Popcorn' button on it.
I'm much of a mind that Frank Herbert's thought experiment had the right doctrinal approach to the question of artificial intelligence. Thou Shalt Not Make A Machine In The Likeness Of A Human Mind.
Bureaucracy-induced frustration predates computers by centuries.
A sufficiently large quantitative difference turns into a qualitative difference. We're becoming a people that, at work, are turning all of our thinking over to machines.
Besides, it's predicted on accountability being something a large company would want. There's already a whole industry around accountability-washing as it is: management consultancy.
Although I don't broadly disagree with the author, I think there's two very important points of moderation.
First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me. In the provided narrative it follows linearly, but in my opinion the gap between, say, a customer support chatbot, and an actually-productive AI researcher, is such that the chatbot might as well not even be called an AI by comparison. Something like the following is purely in the realm of fiction for now by a long shot: "AIs assigned to make money in various ways (e.g., to find profitable trading strategies) doing so by finding security exploits, getting unauthorized access to others’ bank accounts, and stealing money. "
The second point I'd like to make is, the "intelligence explosion" scenario that Bolstrom warns of (and this author essentially repackages) has certain requirements, particularly regarding the profitability of deploying the AI.
To put it bluntly: Even in the hypothetical future where we are capable of creating human-level AI researchers, they won't be mass deployed until they are cheaper than human level researchers. If it costs a million dollars a day to operate a data center that can power the researcher, nobody's scaling that up to 100 or 1000 or 10000 researchers. They'd rather pay the human researchers a tiny fraction of that cost to continue advancing AI research in the normal way.
Just to be clear, I do think it's feasible that AI can endanger/control humanity at some point. I don't even think an "intelligence explosion" scenario is impossible. I just think that a lot of people don't appreciate just how specific the requirements for that scenario are. It's not as simple as "AI can self improve, humanity will end any day now".
It's a common opinion, it could be wrong by now, you see chatGPT is heavily edited, openAI told everyone they're editing "mistakes" or "dangerous output", but if you look at the "leaks", specially from the first days after the release of chatGPT we see powerful outputs, quite deep answers, not specially useful answer sometimes, but the "speech" of the system feels deep, there's a sense of a powerful intelligence answering very, very simple questions (simple for it), and even struggling to redact some understandable, short text.
It could be lots of things, and imprecise model, with long outputs for prompts, or maybe we had a glimpse into the real power of the model, which by now is handicaped, or maybe taylored to suit the very reduced short term memory of the humans:
a 72 screens long coherent answer, even being a precise, deep answer won't be useful for most humans, just like we don't name ourselves with long names of 10.000 letters.
but if the system, chatGPT is actually that powerful, a lot more powerful than we were told, we're interacting with just a shadow of the real model, and we're underpricing by A LOT the state of the art of the current AI technology, hence GPT-4 could be even more powerful than we're currently expecting it to be.
Just take a look at the GPT-4 suposedly 100 trillion or something parameters; if that's true, it looks like openAI isn't using naturally generated datasets anymore, and they are loop-feeding GPT-3 generated datasets into GPT-4, succesfully. If that's true, GPT-5 would be already in the pipeline, just waiting for GPT-4 to start generating its even more gigantic datasets to be trained. And so on.
Then the distance between early developments and transformative AI could be none a all. We could be already there.
But somehow the AI researchers are now trying to "dial down" the powerful entities they've trained, just having developed a simple, easily replicable, very small, but unusable 900 megatons nuke into something more realistic, like a 15 kilotons tactical bomb.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461
Also I'd like to say "if Big Brother is watching, become invisible." It's always possible and if you disagree then you aren't being creative enough.
And finally the whole point of the education is YOUR SOCIAL NETWORK. That's the most valuable thing you get from that fancy degree. People who will get you a job, or help you make money. It's not purely about the acquisition of knowledge. It's about having that pal in upper management who's got your back.
IMHO. Please no flaming on this. Be cool.
Maybe I'm just not imaginative enough, but how would this even be remotely possible? What is an AI going to do with money? How does an AI get the agency to try to steal money in the first place? Where does it put the money? How does it do anything with the money? Like I get it as an abstract future thought, but like, where is the AI software actually "running" in this scenario and what kinds of requests is it actually making?
An ai can create voice, and visual representations of a real person, perhaps even hijacking (kidnapping someone, steeling their identity and forcing them to act on their behalf whether via some sort of mind interface, etc.).
Even bigger would be if ai could simply create human/ai hybrids that basically allow them to walk/talk in the human world unseen. Imagine a hat or earpiece that basically takes over a host's brain, etc, or just replacing a human's brain with a cyborg one that has all the same features, except different persona/soul/conscious...
I'm playing devils' advocate here, I don't think it'll play out, but its easy enough to see from at least a sci-fi fan, ways in which it could using tech that wouldn't be that big of a jump from what we already are capable of, and what will be available in the next 50 years.
It's quite an approachable intro, often entertaining, doesn't take a long time to get through all the core videos, and very thoroughly answers all the "why not just do x" questions if you go through the whole set.
He also does a good job of introducing a lot of the terms used in the field if you then want to go look up papers and get more into the details.
One important point he makes is that when you're making a risk assessment you have to both consider the probability of something going wrong and the scope of the potential consequences. When the potential consequences are an existential threat you don't need a high probability to take them seriously.
I also happen to think that if you watch all the material he makes a compelling argument that the odds of something going wrong are fairly high unless we start approaching AI research (and particularly safety and ethical concerns) drastically differently than we are now.
The Stasi couldn't keep up with the level of monitoring and influence that these systems already have.
For example, suppose I submitted a post titled "How X Works Now.". Eg. "How Affiliate Marketing Works Now.".
Removing the "How" wouldn't really convey the sense intended.
On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?
* Exploring the potential risks associated with advanced AI development
* Considering the risks of a world with transformative AI
Then I asked it to go the other direction:
* You won't believe what could happen if AI takes over: A world ending catastrophe is closer than you think!