AI Safety and the Age of Dislightenment
fast.ai
fast.ai
Here's a random bit of fun prescience:
From Dune, by Frank Herbert:
> JIHAD, BUTLERIAN: (see also Great Revolt) — the crusade against computers, thinking machines, and conscious robots begun in 201 B.G. and concluded in 108 B.G. Its chief commandment remains in the O.C. Bible as “Thou shalt not make a machine in the likeness of a human mind.”
The usual interpretation is something along the lines that AIs took over the thinking for the leaders, making the leaders lazy, impotent, and society fragile and chaotic. This fits with a lot of Herbert's themes.
But, from Children of Dune:
> Three — Planetary feudalism remained in constant danger from a large technical class, but the effects of the Butlerian Jihad continued as a damper on technological excesses. Ixians, Tleilaxu, and a few scattered outer planets were the only possible threat in this regard, and they were planet-vulnerable to the combined wrath of the rest of the Imperium. The Butlerian Jihad would not be undone. Mechanized warfare required a large technical class. The Atreides Imperium had channeled this force into other pursuits. No large technical class existed unwatched. And the Empire remained safely feudalist, naturally, since that was the best social form for spreading over widely dispersed wild frontiers — new planets.
Arguable paints a different picture - that the AI in the hands of the masses would be too threatening to the status quo.
That is 100% what the whole AI panic is about.
Oh certainly. Not for any higher purpose, but remember that OpenAI lost its shit and threatened to leave the moment the EU wanted to look into actually regulating the risks of their product. The driver is money, and little else.
AI people like Sam Altman want a very specific kind of "AI regulation" - a mostly toothless one meant to address the hypothetical of a Skynet (which is honestly still pretty far out there) rather than the more realistic problems that come from their product.
Things like IP law (these LLMs rely on scraping lots of copyrighted data), profiling people (in an age where privacy is important to a lot of people), medical malpractice (I remember someone saying they fed a patient dossier into ChatGPT on this site... Dear Lord I hope that person got shafted by HIPAA) are all genuine problems/risks with LLMs that don't come from the models themselves but from the people using them. Thats the real danger and OpenAI wants absolutely no regulation in that area because it could hurt their bottom line.
Hence why they also almost all tend to follow the beliefs of Elezier Yudkowsky, whose entire life is basically to claim Rokos Basilisk is totally happening and the only option is to give him (and his friends) lots of money to ensure it so you won't be tortured by the AI for not bringing it about.
Crowing about evil AIs is a useful distraction because it plays well populistically (Skynet, Matrix, HAL 9000 are all good examples of "AI out of control" in the public consciousness) and it gives them easy backing from people who don't understand how these things work. The EU wasn't impressed and the AI act they passed addresses some actually meaningful concerns. It remains to see if US Congress/Senate will address it at all or if their business interests reign supreme.
Meanwhile, I'm possibly one of the most Googleable people around (not due to fame, but due to nearly global name uniqueness) and it should be pretty plain to see that I'm a good actor (and kind of a big nerd) and that this might have all been a misunderstanding.
I'm kind of over it, but it is still hugely annoying. And it's terrifying to consider that no matter how anonymous you are on Reddit (and you might have very good reasons to be using an anonymous login), they in theory know exactly who you are.
What I'm saying is that AI could magnify a single human error in the past into an arguably massively unethical action; it could retroactively tie your anonymous actions in the past to your non-anonymous identity in the present. Where there is no escaping your past mistakes and you will always be punished for them, forever, by massive machines without any human intervention, or by systems in power that are arguably unethical and dystopian.
The one thing the Reddit Admin team (actual employees) takes seriously is people making new accounts to post in subreddits where they are banned.
Kill me for having a "porn" Reddit account and a "professional" Reddit account, I guess, plus one to disclose health issues I'd prefer to keep private... And deserve to. I actually enjoy anonymously helping people get through issues that for very good reasons they are also posting anonymously about (think: victims). This mentality ruins that, and I am now prevented from doing that.
Cue Martin Fowler's legendary https://martinfowler.com/articles/bothersome-privacy.html
Your problem is that one of the heads of your hydra caught a ban, after which point everything you do is assumed to be malicious. Given the political nature of what you were banned for, there is no doubt in my mind this is intentional. It's not an overt act of discrimination if they just don't answer the phone after "accidentally" locking you out.
For your own sake, move on. Reddit isn't worth fighting for. It's just a bunch of schoolyard bullies squatting on the playground equipment, throwing rocks at everyone and crying victim. If you're not a member of a protected class (or pretending to be), they don't want you there.
But the one that was just released about AI for autonomous drones, is pretty shocking.
The military side is farther along than I thought, AND, you know what is appearing in a Netflix documentary is old, it isn't the latest things that are still secret.
In any case, why it relates to this article.
We are in a race with other nation states, so all of this discussion about 'curtailing' or limiting AI, is all just grand standing, smoke screen.
Nothing will slow down development because other countries are also developing AI, and everyone firmly believes they have to be first or will be wiped out. Hence no one will slow down.
PS: I don't thik AI will pose an existential risk to our species anytime soon, if ever. But it can already cause damages if used improperly, like any other technology. So it has to be regulated, like any other technology.
Nuclear weapons are a risk to everyone, so the world(US, Russia) did come together in various treaties to limit production, development, etc... So, this could apply to AI also, everyone sees the risk and comes together to curb their us and development.
On other hand:
Nukes are very difficult to make. Iran and North Korea do not abide by the UN resolutions, they are continuing trying to develop them. We are just lucky that is not so easy.
With AI, the barrier to entry is much lower. You can find things that can be weaponized for free on github. It isn't like they are going to bother reading the licenses and not use them.
And for compute power, sure, they have harder time getting the best latest CPU's, and there is an embargo. But still, I think, far easier to get illegal gotten CPU's, than developing Nukes, the barrier is still lower.
Yes, hence the whole “doom”
The existential threat argument is a ridiculous notion to begin with. Pretending open source LLMs don’t already exist makes the argument even more silly.
https://twitter.com/jeremyphoward/status/1678558165712113664
Whether it's a convincing argument is different question, but it's not for lack of knowledge about the state of art in AI.
Why not examine the usage of existing models and judge them on their merits?
But what happens if regulation makes the use of these models more legally dangerous than a few "well regulated" models approved by some sanctioning entity?
That feels like the point of the article to me. If we aren't careful, regulations might have really unwanted impacts.
AI is already increasingly helpful to AI researchers. So one convincing argument for existential risk is that if this increase continues we could see exponential growth in the efficacy of AI research. Given that the goal of AI research is useful intelligence, it seems reasonable to say that an exponential increase in AI research will lead to an at least linear increase in useful intelligence.
Another argument is that there are breakthrough discoveries that will dramatically increase useful intelligence, or dramatically increase the rate of increase. We've had some of these so far, so it seems likely there are others awaiting us. And it's impossible to say with any certainty that there's no chain of such discoveries that doesn't lead to a human or superhuman level of intelligence. All we can do is try to estimate the odds of it happening.
Both arguments seem a pretty clear argument for an existential threat, because as soon as you have even just a human level intelligence integrated that seamlessly with the power of conventional computing, you have something that is much more powerful than most (if not all) human organizations, and once that power exists, there's a non zero risk that it will aim at some objective that is harmful to us, in a comprehensive enough way that it also aims to stop us from being able to stop it.
I wouldn’t say dangerous AI is out of the realm of possibilities, only that we are far from a point technologically where it makes sense to have the discussion. It would be like speculating about the dangers of nuclear energy before discovering the atom.
Our current models do not pose an existential threat to humanity. Period. And yet here we are, discussing the merits of banning open source LLMs. Language modeling is genuinely useful. I’m worried that the AI hysteria will push us toward a future where the best models are needlessly regulated and controlled by corporations.
Incorrect. Power comes with the ability to command resources, not intelligence. The United States military is the most powerful human organization for its ability to marshal the most amount of deadly weapons. Some rinky dink AI company cannot do this.
https://www.theverge.com/2023/7/9/23788741/sarah-silverman-o...
Imagine companies thinking they can scrape all the world's information for free and then package it up and sell it.
Until you get sued into bankruptcy by millions of litigants.
That's what's happening now.
But sure, people will always make these unfalsifiable arguments about some hypothetical doomsday that is far enough in the future that they can never get called out when it doesn't arrive.
That's one of the main business models of the Internet, only usually it's not even scraped. People upload it willingly in exchange for free hosting and free social media.
The other big business model of the Internet is mass surveillance driven targeted advertising.
And if someone hosts user-submitted copyrighted information without a license, the copyright holder can submit a complaint via DCMA and the hoster has safe harbor from liability.
OpenAI is literally scraping copyrighted works, packing it up and selling it without a license. Art, books, magazines, everything. No safe harbor from DMCA for doing that.
A big problem here is that copyright law was a massively problematic thing even before transformer model tech was developed. It has been distorted beyond recognition by a few companies, who have state everything on a broad and permanent copyright rather than the limited one we started out with.
We probably need new definitions in the law, because pretending that training a model on some thing equals copying it isn’t based in reality. It’s an emotional appeal meant to gin up outrage.
This type of interaction is not helpful. It's an argumentative strawman argument. Who said training a model is the same thing as copying a book? Of course it isn't. But who said it had to be?
Here's an idea... read the first five books in the series "A Song of Ice and Fire". Then sit down and write your own version of Book 6 to continue the story and sell it without a license. Guess what's going to happen? You will be sued into bankruptcy.
What OpenAI is doing is lot more similar to that than literally copying things. And it's still wrong and illegal.
It has been distorted beyond recognition by a few companies, who have state everything on a broad and permanent copyright
I agree with you, what Disney and others have done with copyright extensions is immoral and should be illegal. But it's not illegal.
pretending that training a model on some thing equals copying it isn’t based in reality
No, it isn't based in reality. Which is why nobody made the claim. They're packaging up a derivative work and selling it. Don't have to look hard to see examples that this is just as infringing as outright copying.
Pardon me. I did conflate the overall claim that it inherently violates copyright law with a more specific claim (not made) that it "is copying." Since copyright also enumerates the "making derivatives" rights as well as the "copy rights" I acknowledge you have in your argument more than the zero legs to stand on that i implied.
> They're packaging up a derivative work and selling it. Don't have to look hard to see examples that this is just as infringing as outright copying.
This is an interesting claim. It rests on the question of whether the model itself is a derivative work, or if it's a tool (or something between a tool and a trained person).
A photocopier can be used to reproduce ASOIAF and a word processor can be used to create a blatantly derivative work, but I assume we agree that that isn't the problem of Xerox or Microsoft. The derivative works produced with those technologies are the 'illegal' items, not the programs that were used to build them.
If I wrote my own GoT fanfiction, ripping off whole characters, names, and settings, and read my own stories in the privacy of my own home, am I breaking any copyright law? I don't think I would be. I would rightly get in hot water if I tried to sell them, and would probably rightly get in hot water even if I just posted them to Github for free given that I'm distributing the derivative works.
I think using AI tools to generate derivative works could place the user (Not OpenAI, etc) in rightful legal jeopardy if they distribute or sell those works -- on the other hand, if they are simply keeping them for their own personal enjoyment I think it's not that different than if they wrote them themselves. (I also think that rightsholders are acting a little paranoid with those concerns, as though anyone would seriously choose not to buy the latest book or movie or painting only because some poor AI knock-offs exist, but I acknowledge that has little bearing on whether some action is or is not legal.)
I'm not sure I understand this. It is OpenAI that is scraping the copyrighted works, packaging them up into a derivative work and selling access to it.
If a user never enters a prompt asking ChatGPT to create a new ASOIAF book, pieces of those previous copyrighted books are still in OpenAI's model and available for sale by OpenAI.
Chat-GPT the LLM itself is the derivative work that OpenAI is selling access to.
I mean, lets say I am a storywriter and I have an exceptional memory when reading books, and you buy access to talk to me to get story ideas (as a human being, no API). Lets say I also read ASOIAF. Are you telling me that anything I write that mentions winter is now intellectual property of GRRM?
In my eyes your idea of derivative work can fuck right off. Pieces of those of that copyright are also in my mind, but I give no ownership, nor any privilege's to said book writers. IP holders do not get all the benefit of free data in society, then hold the rest of us hostage.
> There are interventions we can make now, including the regulation of “high-risk applications” proposed in the EU AI Act. By regulating applications we focus on real harms and can make those most responsible directly liable. Another useful approach in the AI Act is to regulate disclosure, to ensure that those using models have the information they need to use them appropriately.
We wish. AI is hardware-limited and hardware is not moving fast. We are very, very far from matching raw compute power of the human brain. Robots are even more limited compared to human body.
But that's not an apples-to-apples comparison, as the artificial neurons of neural networks are just a rough approximation of our neurons. They are also connected in a much simpler way too (nicely divided into layers, where each layer can only pass signals to the very next layer), so it could be that it isn't just a matter of how many parameters you have.
AFAIK the biggest limitation of von Neumann computers is memory bandwidth. Brain is moving 10+ PB/s between neurons. This does not count synapse-local and intra-neuron bandwidth.
Assuming you have 200 IQ, if you could make thousands or millions of clones of yourself quite cheaply, you still might not succeed in taking over the world, but it is no longer a laughable idea.
Overall, I’m somewhat ambivalent about the possibility of x-risk, especially if we work reasonably hard to prevent it.
But it shouldn’t be ignored. AI is moving very quickly and it is unclear how powerful its capabilities will be in the next 10-20 years. Of course, there are many other risks presented by AI that we need to stay on top of as well.
Copying AIs just results in effectively higher intelligence (bigger brain). Things would be different with self-replicating robots of course, but duplicating robots is not that fast and one could argue that the robots are dangerous rather than the AI controlling them, because they would be dangerous even if controlled by stupid non-AI program.
Earning money, strategizing, researching, smooth-talking humans, playing politics, and so on. Potentially even improving itself.
These activities could give it power and influence.
Name the next most intelligent animal to humans...
Have humans completely and utterly conquered them?
The intelligence explosion already happened and humans were it. Now you're questioning if it's possible with intelligence explosion 2.0?
I wish I were kidding.
This view is just pure human ego talking.
Dolphins probably think we're fucking morons.
It's not about 200 IQ, it's about when we reach bigger gaps than that. Though yes, I suspect if the 200 IQ guy can copy himself at will, get hardware to do years of 200IQ thinking in an hour, etc. he might succeed.
The hairless monkeys were embodied and violent and they still took millions of years to dominate. The question is whether intelligence on its own provides sufficient advantage to be immediately dangerous. How can you conquer the world by doing "years of 200IQ thinking in an hour"?
I'm pretty neutral about it at the moment, but it's not inconceivable that something with an IQ of 400 and can think 30x faster than you, has no need to sleep and has 500x the working memory of you would be dangerous if it didn't like you.
In fact, your comment and others like it make me wonder if people are just in denial because as I said, it doesn't seem impossible?
How about 500 iq with a direct line into the entire information retrieval and processing system of 7 billion humans, most of whom are already easily manipulated by relatively rudimentary social media ML.
Also, many people seem to be mistaking "artificial intelligence" for "actual intelligence".
You know that these LLMs are not actually intelligent, right? Right???
If an LLM can compose a poem found in no existing book, paint an attractive picture that no one has ever seen, or write a program that has never run before, it's intelligent enough to be called "intelligent."
Conversely, if you can do these things in the absence of any training or outside influence, you're intelligent enough to be called a god. Assuming that this is not the case, don't hold machines to standards you're not willing to hold humans to.
This has to be a candidate for the most unhelpful statement possible. If you are sincere about it, then why not apply it to every word in your reply and see where that leaves you?
Humans certainly have flaws when it comes to this. I've heard some discussions about the success and failures of AI in this regard. Can someone in this domain elaborate on the current state-of-the-art performance in this regard?
2. LLMs see positive transfer in multi lingual capability. For example, an LLM trained on 500B tokens of English and 10B tokens of French will not speak in french like a model trained on only 10B tokens of French. What will happen is that the model will be nearly as competent in French as it is in English https://arxiv.org/abs/2108.13349
3. Language models of code reason better even if the benchmarks have nothing to do with code.
(Apologies to any linguists. Please correct anything above if I'm off).
As for the other post, degraded performances are highly non trivial still. Some aren't actually poor, just worse.
Even the authors admit humans would see degraded performance on counterfactuals unless given "enough time to reason and revise", something they don't try to do with GPT-4.
Think about it. Do you genuinely belief you would score as accurately on a multiplication arithmetic test taken in base 8 ?
No, but I believe this is a different question. I think the more relevant question is whether a human can (even with the caveat of needing more time to reason about it). The larger question for a LLM is whether it can answer it at all and interpret why, without additional training data.
The paper seems to point that the ability of LLM to transfer is related to proximity to the default case. E.g., if default is base 10, is better at base 9 than base 2. I would interpret that as indicating more simple pattern recognition than deductive reasoning. The implication being that real transference is more dependent on the latter.
arithmetic but in a different base is one of the counterfactual examples in the paper. That's why i mentioned that. and yes it can answer them with worse performance.
You can juice arithmetic performance as is with algorithmic instructions. https://arxiv.org/abs/2211.09066. I see no reason the same for other bases wouldn't work.
Even if you gave a human substantial time for it (say a week of study), i believe he/she almost certainly reach the same accuracy unless he had access to specific instructions for working in base 8 he/she could call upon when taking the test.
I know, that's why I referenced the proximity of bases seemingly being important to the LLM. I think this is what differentiates it.
>and yes it can answer them with worse performance.
It's accuracy is dependent on proximity to it's training set (going back to my original point). I think that points to a different mechanism than humans and that's what my last post was focusing on.
I think we agree that humans would do less well in most other bases than base-10. But that side-steps the point I was making. Will humans do worse in base-3 than base-9? I doubt it, but according the the article, it's reasonable to assume the LLM would be progressively worse. That, IMO, is an indicator that something different is going on. I.e., humans are deriving principles to work from rather than just pattern recognition. Those principles can be modified on-the-fly to adjust to novel circumstances without needing additional training data. Humans are using reasoning in addition to pattern recognition.
This is probably a clunky example, but I'll try. Suppose an autonomous vehicle is trained to recognize that when a ball rolls into the street, it needs to slow down or stop because a child may not be far behind. A human can infer that seeing a kite blow into the street may signal the same response, even though they've never witnessed a kite blow into the street. The question is: can the autonomous vehicle infer the same? (This shouldn't be conflated with the general case of "see object obstructing the street and slow down/stop." The case I'm drawing here specifically adjusts the risk by the nature of the object being a child's toy. So, can the AV not only recognize the object as a kite but also adjust the risk accordingly?) I think one of the possible pitfalls is that we solve a more simple problem like image/pattern recognition and conflate it to a more difficult problem set being solved.
Circling back to the original point, one guess is that it's not understanding context as much as merely matching patterns really, really well. That can be incredibly useful but it may be something different than what's going on in our heads and maybe would should be careful not to conflate the two. Or, it's possible that all we're doing is also matching patterns in context, and eventually LLM will get there too.
I genuinely don't see how that would be a reasonable assumption.
>Will humans do worse in base-3 than base-9?
Why not? If you haven't learnt base 3 but you have base 9 you'll do poorer on it.
>That, IMO, is an indicator that something different is going on.
Whether something different is going on is about as relevant as the question of whether submarines swim or plans fly or cars run.
>I.e., humans are deriving principles to work from rather than just pattern recognition.
Not really. Nearly all your brain does with sense data is predict what it should be and adjust your perception to fit. You can mold these predictions implicitly with your experiences but you're not deriving anything from first principles.
>This is probably a clunky example, but I'll try. Suppose an autonomous vehicle is trained to recognize that when a ball rolls into the street, it needs to slow down or stop because a child may not be far behind. A human can infer that seeing a kite blow into the street may signal the same response, even though they've never witnessed a kite blow into the street. The question is: can the autonomous vehicle infer the same? (This shouldn't be conflated with the general case of "see object obstructing the street and slow down/stop." The case I'm drawing here specifically adjusts the risk by the nature of the object being a child's toy. So, can the AV not only recognize the object as a kite but also adjust the risk accordingly?) I think one of the possible pitfalls is that we solve a more simple problem like image/pattern recognition and conflate it to a more difficult problem set being solved.
Casual reasoning ? all evidence points to LLMs being more than capable of that https://arxiv.org/abs/2305.00050
It's not an assumption. It's literally based on the results of your own reference:
>LM performance generally decreases monotonically with the distance
If you can't be bothered to read your own reference, I don't think additional conversation is worthwhile because it becomes apparent that it's more dogmatic than reasoned.
Your newest link is not really supportive of your "all evidence" claim. It goes into further detail about how LLM can have high accuracy while also making simple, unpredictable mistakes. That's not good evidence of a robust causal model that can extrapolate knowledge to other contexts. If I didn't know better, I'd assume you could just as well be a chat bot who only reads abstracts and replies in an overconfident manner.
Human performance generally decreases with level of exposure so I figured you were talking about something else. Guess not.
>I don't think additional conversation is worthwhile because it becomes apparent that it's more dogmatic than reasoned
By all means, end the conversation whenever you wish.
>It goes into further detail about how LLM can have high accuracy while also making simple, unpredictable mistakes.
I'm well aware. So? Weird failure modes are expected. Humans make simple, unpredictable mistakes that don't make any sense without the lens of evolutionary biology. LLMs will have odd failure modes regardless of whether it's the "real deak" or not, either adopted from the data or from the training scheme itself.
>If I didn't know better, I'd assume you could just as well be a chat bot who only reads abstracts and replies in an overconfident manner.
Now you're getting it. Think on that.
Are you saying that as humans get more experience, they perform worse? I disagree, but irrespective of that point it’s wild that you can have this many responses while still completely bypassing the entire point I was making.
I don’t think most would argue that performance increases with experience. The point is how well can the performance be maintained when there is little or no exposure. Because that implies principled reasoning rather than simple pattern mapping. That is the entire through line behind my comments regarding context dependent language, novel driving scenarios, etc.
>Think on that
In the context of the above, I don’t think this is nearly as strong of a point as you seem to think it is. There nothing novel about a text-based discussion.
If that child had the basic teaching most children do (little exposure) then a quiz will result in much worse performance than a base 10 equivalent test. This is very simple. I don't know what else to tell you here.
2. You must understand that a human driver that stops because a kite suddenly comes across the road doesn't do so because of any kite>child>must not hurt reasoning. Your brain doesn't even process information that quickly. The human driver stops (or perhaps he/she doesn't) and then rationalizes a reason for the decision after the fact. Humans are very good at doing this sort of thing. Except that this rationalization might not have anything at all to do with what you believe to be "truth". Just because you think or believe it is so doesn't actually mean it is so. Choices shape preferences just as much as the inverse. For all anyone knows and indeed most likely, "child" didn't even enter the equation untill well after the fact.
Now if you're asking whether LLMs a matter of principle can infere/grok these sort of casual relationships between different "objects" then yes as far as anyone is able to test.
Regardless, it still misses the point. I’ve never been explicitly exposed to base-72, yet I can reason my way through it. I would argue my performance wouldn’t be any different than base-82. So I can transfer basic principles. What the LLM result you referenced shows is that it is not learning basic principles. It sure seems like you just read the abstract and ran with it.
As far as the psychology of decision making, again, I think you're speaking with greater confidence than is warranted. In time critical examples, I’m inclined to agree. And there’s certainly some notable psychologists who would expand it beyond snap judgments. But there are also some notable psychologists who tend to disagree. It’s not a settled science, despite your confidence. But again, that’s getting stuck in the limitations of the example and missing the forest for the trees. The point is not in whether decisions are made consciously or subconsciously, but rather how learning can be inferred from previous experience and transferred to novel experiences. Whether this happens consciously or not is besides the point. And you are further going down what I was explicitly taking against: confusing image/pattern recognition for contextual reason. You can see this in the relatively recent Go issue; any human could see what the issue was because they understand the contextual reasoning of the game but the AI could not and was fooled by a novel strategy. The points I’ve been making have completely flew over your head to the point where you’re shoehorning in a completely different conversation.
I guess so. I've never meant to imply greater exposure leads to worse outcomes.
>I would argue my performance wouldn’t be any different than base-82.
Even if that were true and i don't know that i agree, the authors of that paper make no attempt to test in circumstances that might make this true for LLMs as it might for people. So the paper is not evidence of the claim (no basic principles) either way. For example, i reckon your performance on the proceeding 82 test will be better if taken a immediately after than if taken weeks or months later. So surrounding context is important even if you're right.
>What the LLM result you referenced shows is that it is not learning basic principles.
I disagree here and i've explained why.
>You can see this in the relatively recent Go issue; any human could see what the issue was because they understand the contextual reasoning of the game but the AI could not and was fooled by a novel strategy.
You're talking about this ? https://www.zmescience.com/future/a-human-just-defeated-an-a...
KataGo taught itself to play go by explicitly deprioritizing “losing” strategies. This means it didn’t play many amateur strategies because they were lost early in the training. This is hard for a human to understand because humans all generally share a learning curve going from beginning to amateur to expert. So all humans have more experience with “losing” techniques. Basically what I’m saying is, it might be that the training scheme of this AI explicitly prioritized having little understanding of these specific tactics, which is different than not having any understanding.
This circles back to the point I made earlier. Having failure modes humans don't or won't understand or have is not the same as a lack of "true understanding".
We have no clue what "basic principles" actually are on the low level. The less inductive bias we try to shoehorn into models, the better performing they become. Models literally tend to perform worse the more we try to bake "basic principles" in. So presence of an odd failure mode we *think* belies a lack of "basic principles" is not necessarily evidence of a lack of it.
>The points I’ve been making have completely flew over your head to the point where you’re shoehorning in a completely different conversation.
You're convinced it's just "very good pattern matching", whatever that means. I disagree.
E.g., racism/sexism/...most -'isms' appear to be a general heuristics that help us make quick judgements. But we can also our decision-making process by reverting to basic principles, like the idea that humans have equal moral worth regardless of skin tone or gender. AI can even mimic these mitigations, but you haven't convinced me that it can fundamentally change away from it's training set based on an understanding of basic principles.
As for the Go example, a novice would be able to identify that somebody is drawing a circle around it's pieces; your link even states this. But you recharacterizing this as a specific strategy is weird when that strategy causes you to lose the game. It misses the entire meaning of strategy. We see the limitations of AI in it's reliance to training data from autonomous vehicles to healthcare. They range from the serious (cancer detection) to the humorous (Marines overtaking robots by hiding in boxes like in Metal Gear). The paper you referenced similarly shows it is reliant on proximity to the training set, rather than actually understanding the underlying principles.
Humans don’t have a grasp of the “principles of reasoning” and as such are incapable of distinguishing “true”, "different" or “heuristic” assuming such a distinction is even meaningful. Where you are convinced of “faulty shortcut”, I simply think “different”. Multiple ways to skin a cat. a plane's flight is as "true" as any bird. There's no "faulty shortcut" even when it fails in ways a bird will not.
You say humans are "true" and LLMs are not but you base it on factors that can be probed in humans as well so to me, your argument simply falls apart. This is where our divide stems from.
>I don't think you've provided evidence that AI can have a principled understanding while we can show that humans can.
What would be evidence to you? Let’s leave conjecture and assumptions. What evaluation exist that demonstrate this “principled understanding” in humans? and how would we create an equitable test in LLMs?
>a novice would be able to identify that somebody is drawing a circle around it's pieces; your link even states this. But you recharacterizing this as a specific strategy is weird when that strategy causes you to lose the game.
You misunderstand. I did not characterize this as a specific “strategy”. Not only do modern Go systems not learn like humans, but they also don’t learn from human data at all. KataGo didn’t create a heuristic to play like a human because it didn’t even see humans play.
>The paper you referenced similarly shows it is reliant on proximity to the training set, rather than actually understanding the underlying principles.
Even the authors make it clear this isn’t necessarily the bridge to take so it’s odd to see you die on this hill.
The counterfactual of syntax is Finding the main subject and verb of something like “Think are the best LMs they.” in verb-obj-subj order (they, think) instead of “They think LMs are the best.” in subj-verb-obj order (they, think). LLMs are not being trained on text like the former to any significant degree if at all yet the performance is fairly close. So what, it doesn’t “underlying principles of syntax” but still manages that ?
The problem is that you take a fairly reasonable conclusion from these experiments. I.e LLMs can/often also rely on narrow, non-transferable procedures for task-solving and proceed to jump the shark from there.
>but you haven't convinced me that it can fundamentally change away from it's training set based on an understanding of basic principles.
We see language models create novel functioning protein structure after training, no folding necessary.
https://www.researchgate.net/publication/367453911_Large_lan... So does it still not understand the “basic principles of protein structures”?
"we do not expect our language model to generate proteins that belong to a completely different distribution or domain"
So, no, I do not think it displays a fundamental understanding.
>What would be evidence to you?
We've already discussed this ad nauseum. Like all science, there is no definitive answer. However, when the data shows evidence that something like proximity to training data is predictive of performance, it's seems more like evidence of learning heuristics and not underlying principles.
Now, I'm open to the idea that humans just have a deeper level of heuristics rather than principled understanding. If that's the case, it's just a difference of degree rather than type. But I don't think that's a fruitful discussion because it may not be testable/provable so I would classify it as philosophy more than anything else and certainly not worthy of the confidence that you're speaking with.
Good thing they don't make sweeping declarations or say anything about that meaning narrow learning without transfer. Jumping the shark yet again.
https://www.pnas.org/doi/full/10.1073/pnas.2016239118
>We find that without prior knowledge, information emerges in the learned representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity. We show the learned representations are useful across benchmarks for remote homology detection, prediction of secondary structure, long-range residue–residue contacts, and mutational effect.
From the sequences of just the proteins alone, Language Models learn underlying properties that transfer to a wide variety of use cases. So yes, they understand proteins in any definition that has any meaning.
>Good thing they don't make sweeping declarations or say anything about that meaning narrow learning without transfer.
That's exactly what that previous quote means. Did you read the methodology? They train on a universal training set and then have to tune it using a closely related training set for it to work. In other words, the first step is not good enough to be transferrable and needs to be fine tuned. In that context, the quote implies the fine tuning pushes the model away from a generalizable one into a narrow model that no longer works outside that specific application. Apropos to this entire discussion, it means it doesn't perform well in novel domains. If it could truly "understand proteins in any definition", it wouldn't need to be retrained for each application. The word you used ('any') literally means "without specification"; the model needs to be specifically tuned to the protein family of interest.
You are quoting an entirely different publication in your response. You should use the paper from which I quoted to refute my statement, otherwise this is the definition of cherry picking. Can you explain why the two studies came to different conclusions? It sure seems like you're not reading the work to learn and instead just grasping at straws to be "right." I have zero interest in having a conversation where someone just jumps from one abstract to another just to argue rather than adding anything of substance.
This is why it's mostly meaningless to for a LLM to pass the bar, but not meaningless for a human to do so. We (rightly, for the most part) assume that a human who passes the bar can transfer those skills into unique and novel situations. We can't make that assumption for LLMs, because they are lacking adaptability that is needed for true intelligence.
If you took arithmetic tests in base 8, you wouldn't reach the same accuracy either.
The word was made up to cover a range of cognitive abilities that humans and animals (to varying degrees) possess. And we're gradually figuring out how to design machines with similar abilities. The general idea of intelligence being you can figure out how to do things that aren't just instinctual. a generalized intelligence can do this across any number of domains without some fixed limit.
So you're saying humans, collectively, are gods? After all, as a species we started with nothing - no "clean" training data, no paintings to replicate, not even language itself. And here we are - arguing about whether one creation of ours, trained on a bunch of other stuff we created, is as intelligent as us.
So is all human knowledge. They're all things we "made up" to describe the world around us. That's a really cheap way to reduce something. There's a mountain of psychological literature that intends to describe and measure intelligence. Saying it doesn't mean much of anything is a bit of a postmodern hot take.
> If an LLM can compose a poem found in no existing book, paint an attractive picture that no one has ever seen, or write a program that has never run before, it's intelligent enough to be called "intelligent."
Painting pictures and composing poems aren't good ways to measure general intelligence. That's creativity. They're correlated but not the same.
There are specific tests for measuring general intelligence, and there's absolutely no way current gen LLM would score highly on them because they don't even accept visual input. Yes I've seen that they can score highly on verbal-only tests. That's obvious anyway because they have seen, written down, and have access to the answers during the test.
Even if they did accept visual input they would only be able to solve problems for which they've previously been given the answers (trained on the dataset). Even then I'd have my doubts given how terribly wrong ChatGPT4 is when I ask it to produce a very simple Kubernetes manifest which has a strictly defined spec.
One problem is that ChatGPT4 has been degrading over time -- and no, I don't GAF about any opinions or assurances to the contrary. So we're seeing more and more people look at it for the first time and ask what the big deal is. Maybe the Code Interpreter feature will reverse that trend, we'll see.
Those of us who were around (read: paying $20/month) when they first enabled GPT4 support are left waving our hands fecklessly, muttering "Yeah, but you should've seen it back in the old days, back in March of '23. Now you kids get off my lawn."
An LLM can write a poem that fools me into thinking it’s good, but not anyone who reads poetry as a hobby. Poetry is inspired in a way that’s hard to replicate with a prompt.
Artists also take inspiration from other artists, but the “distance” between their influences and their work is so much greater than the distance between an LLM’s that I don’t fault anyone who think it’ll take time for LLMs to catch up.
And you don't grasp the fundamentally-incremental nature of this point?
What about version 5? 6? 10?
quality is required.
I think once you give AI working limbs and the ability to manipulate matter like humans, that's concerning, because if the AI becomes a runaway AI with no upper limits on the types of behavior allowed, we get into Paperclip Maximizer[0] territory very fast.
Everyone wants to pose as the front(wo)man of that innovation, thought-leaders. Whatever.
It's all marketing, for their own careers, their own personal agenda.
The whole text looks like written by ChatGPT in a prompt like "Write about AI safety and relates it to the Age of Reason".
This is a good reason why AI should replace us, so we no longer have human beings writing that stuff to self-promote themselves.
The funniest part is in the end of the article:
"Eric Ries has been my close collaborator throughout the development of this article"
So the guy who wrote "The lean startup" is helping him with that article. drops mic
They've had considerable success getting traction for these views with politicians in Europe, and that's the reason you see so much of this in the media currently. I'm hoping more sane voices will soon get organized, because significant parts of this are reminiscent of a well-funded doomsday cult.
I think this article is an early attempt at creating a politically viable counterpoint.
Got to hang out with some "hotshot" Facebook developers after hours. I remember Lee Byron (of React, GraphQL, etc) stating very confidently that the roads would be teeming with self driving cars by 2020.
Maybe L2 systems. But those are just a driver assistance, the driver must supervise and intervene.
Actual L4/L5 systems haven't killed anyone as far as I know, except for the Uber accident, and even then the driver was supposed to be supervising it, I'm not sure if it counts as L4/L5
https://arxiv.org/abs/2303.11366
GPT-4's emotional intelligence seems to be very high by the looks of things
https://arxiv.org/abs/2304.11490
Empathy =/ Intelligence
Isn’t that a metric we use to determine the intelligence of animals and such?
Does GPT love its creators the way we love our parents? Does it mourn a loss the way we do?
Elephants are known to be of the more intelligent animals and apparently they mourn loss as well. So apparently there is something there linked with intelligence.
Maybe it's not directly intelligence related, but what makes a human have a spontaneous emotional response like laughter for instance?
The self-reflection link you sent isn't what I was thinking.
I was thinking about things like self reflection on my existence. Like am I happy doing what I'm doing? Do I feel I am making a positive difference in the world? If not what should I start doing to work toward becoming what I want to be?
The fact that I want to be something seems different. Nobody tells me what I want to be. But we still need to tell computers an awful lot about what they should be and what they should want to be. When will AI be able to decide for itself what it wants to be and what it wants to do? And to do all that sort of self reflection on it's own without us specifying a reward system that "makes" it want to do that? I don't strive to become a better in the world person for some reward.
> Isn’t that a metric we use to determine the intelligence of animals and such?
Depends what you mean by those words.
"Do they pass a mirror test" is, as far as I know, the best idea anyone's had so far for self awareness, and that's trivial to hard-code as a layer on top of whatever AI you have, so it's probably useless for this.
> Does GPT love its creators the way we love our parents? Does it mourn a loss the way we do?
I'd be very surprised if it did, given it wasn't meant to, but then again it wasn't "meant" to be able to create unit tests for a Swift JSON parser when instructed in German either, and yet it can.
Trouble is… how would you test for it? Do we have more than the vaguest idea how emotions work in our own heads to compare the AI against, or are we mysteries unto ourselves?
Or can we only guess the inner workings from the observable behaviours, like we do with elephants? But "observable behaviour" is something that a VHS can pass if it's playing back on a TV, and we shouldn't want to count that. Is an LLM more like a VHS tape or like an elephant? I don't know.
LLMs are certainly alien to how we think; as it's been explained to me, I think it should be thought of as a ferret that was made immortal, then made to spend a few tens of thousands of years experiencing random webpages where each token is represented as the direct stimulation of an olfactory nerve, and it is given rewards and punishments based on if it correctly imagines the next scent.
> I don't strive to become a better in the world person for some reward.
Feeling good about yourself is a reward, for most of us.
But I think this is a big difference between us and them: as I understand it, most AI have only one reward in training, while humans have many which vary over our lives — as an infant it may be pain va. food or smiles vs. loud noises, as a child it may be parental feedback, as an adolescent it may be peer pressure and sex, as a parent it may be the laughter vs. the cries of your child… but that's pop psychology on my part.
There’s an argument that the vast majority of human-generated research isn’t really “novel” but just derivative of other ideas. I’m not so sure ChatGPT couldn’t combine existing ideas to come up with something “novel” just like humans. I think there’s a case that it already comes up with creative, novel solutions in the drug space.
Can a submarine swim?
Does it matter if an LLM — which is far from the only kind of AI, and trivial to include in another more complex AI — does or doesn't match your definition of "intelligent" when it can write fluently in any language (human and code), and also getting near top-of-class results in the Bar exam and the biology olympiad?
Corporations — sometimes used as another example of an unaligned optimising agent — can't drive cars, they have to hire humans to do that for them, but corporations can still be prosecuted for dangerous chemicals (3M); and AI, just like other computer programs, can have dangerous output that you should not rely on (e.g. Thule early warning radar incident where someone forgot to tell it that it's OK for the moon to not respond to IFF pings and it nearly started WW3).
This is entirely false, sounds like Andersen that seems incapable of understanding that agency and goals will be a thing.
Nor do they now, of course.
We should focus more on the making infrastructure safe from abuse and not how to cripple AI models for them to not realize how to do harm. More like 'How to ensure humans don't drop atomic bombs on each other' rather than 'How to make atomic bombs safer to drop on people'.
Fly some airplanes into some skyscrapers, etc.
Humans are too fucking lazy to actually accomplish this. We hook more and more internet connected systems to things with a digitally controlled motor every day with things like "#disable in production, set for testing, disabling 2FA to..." that then get pushed to the customer.
Can you prove this?
https://www.lesswrong.com/posts/6untaSPpsocmkS7Z3/ways-i-exp...
> Model licensing & surveillance will likely be counterproductive by concentrating power in unsustainable ways
The first refers to all regulation, while the second is talking about a specific attempt to regulate.
This is like saying "I don't like beef" and "we should ban all meat" are the same statements. They're clearly not, the scope is vastly different.
The capabilities of current generation AI—LLMs + audio/image/video generative models—are far enough already that wide-scale distribution is extremely dangerous.
Social media allowed for broadcasting from 1:many and 1:1 with troll armies, but LLMs are on a whole other planet of misinformation. A scalable natural language interface can target people at the one-to-one conversation level and generate misinformation ad hoc across text, voice, images, and video.
The trouble with this approach is that harm will rack up way faster than liability can catch up, with irreversible and sometimes immeasurable long term effects. We have yet to contend with the downstream effects of social media and engagement hacking, for example, and suing Facebook out of existence wouldn't put a dent in it.
Current generation AI has enough capabilities to swing the 2024 US presidential election. It can be used today for ransom pleas trained on your childrens' instagram posts. It doesn't seem farfetched that AI could start a war. Not because it's SkyNet, but because we've put together the tools to influence already-powerful-enough human forces into civilization-level catastrophe.
The only reason an LLM would even have such access to people to do the hyperbolic damage you claim is thru ad networks and social media anyways - yet you are calling to behead public access to tech instead of regulating propaganda platforms. Both seem facile but your selection is the one that seems subservient
I wasn't convinced, but I did the maths and, well, looks like you're right:
Hiring people in to read and respond in a politically biased way to each person in the USA would take, let's say 2 messages per day for a month, so 60 messages per person, let's say 100 million actual voters you care about so 6 billion messages, let's say 15 seconds per response so 25 million hours, let's say you're hiring Kenyans because I happen to have the GDP/capita and workforce participation rate to hand, so it's about 2 USD/hour on average, so 50 million USD.
With GPT-4, and let's approximate each message as 100 input tokens and 100 output tokens, that's 0.1 * ($0.03 + $0.06) = $0.009, times the same 6e9 messages = 54 million USD.
So, yeah, looks like you're right.
3.5 is cheaper, but tends to be noticeable; from what I've been hearing (fast moving target and all that) the OSS LLMs tend to be 90% of the quality of 3.5.
The battle for the future isn't attention, it's intimacy. LLMs will up the game from broadcast propaganda to individual relationships via digital medium. It will be a two way communication that profiles you and feeds your biases in a much more direct way than even the social media giants themselves have accomplished.
(I still haven't used 4 properly, so possibly? And given the hypothetical situation under discussion, could it have if it had not been directed not to?)
You're conflating a communications channel with a new technology that enables abuse (as well as positive outcomes) at a previously-unimagined scale.
Is it not facile and subservient to default to the ostrich algorithm for new and world-changing technologies?
I'm really not. Do you genuinely think that the last half of your sentence only applies to LLMs? A "technology that enables abuse (as well as positive outcomes) at a previously-unimagined scale" is literally a perfect description of social media platforms. LLMs virtually don't move the needle, look at the amount of propaganda and bots already on the web today.
I think that implying I am burying my head in the sand about LLMs by claiming they shouldn't be restricted to megacorps who already have demonstrated their willingness to engage in unethical activities at mass scale is fucking ridiculous too. But I do welcome your continued support of regulating AI in favor of corporations, it's a clear sign to never engage with you
I'd say you'll be incorrect in the future. We'll have new abuses by both corporations and propaganda bots in the near future.
More and more of us will find 'digital friends' that creep into our lives and talk to us on line, gain our trust, make us laugh, and tell us that 'unions are bad', or whatever the propaganda of the day is. But it's the one on one relationships with history that will make them different than the bots that came before.
That would just mean automating the job of the secret police. And those guys have unions already!
Is this intentional ignorance? The reason I use Facebook as the analogy is because Facebook has an insane amount of your history, and isn't afraid to sell any single aspect of that to advertisers. The interface some chatbot-controlling megacorp would expose to propagandists would hardly be different, the history is unlikely to be much stronger, and the network effect seems hard to match. Less a step-change than an iteration.
Power imbalance is not 'much worse' than extinction.
It's very clear how people start with the conclusion - "openess and progress have been good and will keep being the best choice" and then just contort their thinking and arguments to match the conclusion without really critically examining the counter arguments.
Okay, GLHF.
2 years ago I thought we were decades away from general purpose AI, this is coming from a guy who implemented transformer models on day 5. My time estimates have been proven very wrong.
I'm equally worried about the value of white collar labour dropping to near zero in my lifetime, and the ultimate centralization of power. The movie Elysium seems less and less science fiction every day.
I am happy politicians and think-tanks are taking this seriously, you should be too.
Humans _hate_ being useless, and we were already running into surplus human issues last decade in certain parts of the world.
I foresee a depression/suicide spiral never before seen in our history unless we do something radical
Watching news can distort your perspective very quickly.
The point is people that believe the world will be stable in the future are more apt to build a stable future. If everyone is watching the news and they believe the future will be unstable, then the future will become unstable as a self fulfilling prophecy.
A lot of people that retire from their careers pursue other interests that fulfill their needs, maybe it will be the era of amateur artists everywhere. Others prefer to manually do tasks that can be automated.
Others could engage in cooperation with robots because humans will always have creative desires.
There are any number of problems that have to be solved together. Giving the capital class hyper powerful AI robots so they can own the world while everyone else suffers is an AI risk.
I mean I enjoy my work, for the most part anyway. But if I could spend all my time with my family and hobbies I see that as an improvement rather then something to be depressed about.
The domains where AI experts can beat domain experts is certainly growing, but I don't see how you can get from that to a claim of general AI. I certainly don't see how you can get there from the recent LLMs in particular which can't beat domain intermediates at anything.
The transformer model is one more incremental improvement that made the language problem tractable. This has only captured the public interest because the language problem is so much more flashy and easy to understand than something like protein folding.
I would be remiss in saying they are anywhere near average human capability in most areas, but I do worry that my estimation capability is off and that we're closer to the concave part of the S curve than the convex. It just takes a couple more breakthroughs in model/dataset design
* Predicting the physical geometry of proteins
* Playing Starcraft
* Playing Go
* Trading stocks
* Controlling motors to make a bipedal robot walk
* Analysing data from particle accelerators
* Analysing data from telescopes
* Detecting cancer in medical images
These are all things that AI can do today. Are you suggesting that we are near a paradigm shift where a single AI system will ve competent in all of these domains? And will further be simmilarly competent in novel domains for which it has not been designed?
It's not just about language. Language models can predict novel protein structures, no folding required. https://www.nature.com/articles/s41587-022-01618-2
>What is the roadmap to get to general purpose AI, and what is the proof that we are close?
What testable definition of General Intelligence does GPT-4 fail that a big chunk of humans don't also fail ?
(Shrug) If it turns out that humans have a higher purpose than doing a robot's job badly, I don't see the downside to letting the robots do the job. I'm all for capitalism, but it's an ideology, not a religion. If there's something better on the horizon, let's help it along.
ML looks like it is probably going to be the next step in human evolution by punctuated equilibrium. And it's about time.
What we've discovered is that the first model breakthroughs have been able to largely demolish that dream. As a writer I can tell you that writing with GPT4 today provides me with superpowers that non- accelerated authors could come close to; and theres no plausible way to distinguish our two works.
I'm more worried about the coming depression / suicide spiral (which were already in by the way) because of human uselessness than I am of terminators coming to farm us.
What's your working definition of "general purpose AI"? What does it include? What does it not include?
Does the AI have to be smart enough not to very often state incorrect information? LLMs still are unimpressive to me from that front.
I asked ChatGPT to explain the usage of the term "Wonder Kid" in the popular Apple TV+ series Ted Lasso, and which character it's tied with. It gave me confidently-stated incorrect answers multiple times until I told it the answer. (At which point it stated the correct answer with an apology.)
This doesn't feel close to "general purpose AI". Feels like we're meeting the limits of LLMs and finding they're not a magic AI bullet.
Textbooks are all you need is another extremely interesting area of focus.
Improved correctness and a general reduction in hallucinations I predict will end up hitting OSS models by EOY
John Schulman from OpenAI recently gave a talk about the hallucination and uncertainty issues [1]. He claimed that models already have information about their uncertainty but that it remains an open problem as to how to express that uncertainty in natural language. One of the big issues with trying to prevent the model from hallucinating is that you can err too much on the side of caution, causing the model to lie about things it actually does know the correct answer to.
Oh shit, we've developed digital salespeople...
This is more utopian than communism even.
"There will still be Bad Guys looking to use them to hurt others or unjustly enrich themselves. But most people are not Bad Guys"
Failure to understand that even one by guy can kill everyone with a sufficiently advanced AGI