Religious scholars met with Anthropic
nytimes.com
nytimes.com
When you’re more worried about this sort of stuff vs generating profit then the history books strongly indicate that’s usually not a good omen of what’s coming next.
Also, even if the 20 dollar subs disappear, that just means regular folks can't afford AI. API pricing is still a great bargain for businesses replacing generic office workers.
Capitalists want to pay their workers as little as possible or even not have workers and pocket close to 100% of the profits since forever. Whether they'll be able to do it this time is yet to be seen. Putting the tech itself aside, which is far from good enough yet imo, the only way to avoid a massive social backslash would be to use some of the additional profit they're able to keep because of AI for social spending. And I just don't think they realize yet that it's probably in their interest to do so if they want to replace 'human resources' with AI.
They are clearly smart/dumb enough to think pushing labor replacement to 20%+++ unemployment levels would have no consequences for them personally. Maybe with their private jets, private islands, doomsday shelters, etc they are right.
Seems like a bit of a gamble to me, or probably anyone who has read a bit of history. But I’m not a billionaire.
Have you tried using "computer use" models? They just started to get good in the last few months. They can comfortably use GUIs, click the right buttons, enter the data faster etc. This is very significant and my programmer friends often don't appreciate the earth shattering consequences. The vast majority of software in such applications is legacy stuff or proprietary and does not and likely will never have a programmatic API. The only way to interact with it is via GUI. And an AI working that GUI can be deployed unilaterally by the customer of the software company who made the GUI program. This makes adoption and automation much easier. My prediction is that within 1 year (maybe 2) GUI-using AI will be much more powerful and it will be everywhere (of course it doesn't need to be displayed on a screen, it will do its virtual clicks based on a virtual screen buffer rendering in an isolated virtual machine, along with countless other agents in parallel).
There will be huge controversy around it, because this will be the first instance that regular people will see AI do work that resembles their day job all over social media. There will be panic. There will be new license terms saying only humans are allowed to operate the GUI, and AI use is either forbidden or costs extra. But this will not be checkable properly. Even when forbidden, some employees will pay from their own pockets to have cheap AI do the job instead of them.
Something very similar happened with the late 90s telecoms bubble, which collapsed a little before the dot com bust. There was a very nice living to be made by many small/mid-sized providers who were able to feed on the remains for about a decade afterwards.
1. Datacenter builders are all up to their eyeballs in debt. The companies serving open weight models are getting their GPUs and power from somewhere, and we can infer that some of that is coming from the new build datacenters. But are those financially sustainable?
2. They are relying on subsidized giveaways of the underlying models by Chinese firms. But it's hard to see how that can be sustainable: it's being done for a reason and that reason is probably not commercial. If and when China stops doing this the open weight serving companies will get steadily less competitive with every passing month as the knowledge in the weights becomes obsolete.
Opinions are my own, etc.
Commodified products competing on price does not bode well for valuations.
Also, "circular financing" is a meme. A private company cannot create money, there is no circularity. NVIDIA may buy services from OpenAI and also sell them GPUs. If a lumberjack buys eggs from his neighbor and also sells him wood to improve his chicken coop, that's not circular financing, just business happening in both directions.
And users will make do with what their newly bought cheap hardware will run, which I think is going to be more than enough for everyone that doesn't need to solve another millenium problem.
The use cases LLMs are useful for without lots of RF/fine tuning proves more limited than the expansive views currently held.
We get a generalized productivity boost the way the internet gave us, without any one sector or company being a winner take all of that productivity gain.
We’ve seen this movie before. It’s possible for new tech to be both amazing and revolutionary and a business disaster both at the same time.
What did stop is investment, and they stopped running some lines, and some stuff was nationalized.
I doubt we'll see nationalizations because that's not America's style (thank goodness!) but if the tide goes out on AI funding we should expect to see much longer gaps between model releases, people complaining that model knowledge is getting stale because there wasn't a pretraining run for long enough, price rises, capacity bottlenecks that go unaddressed, few if any new product launches, maybe some mergers and layoffs. Some labs will quit the market. The market might consolidate down to just two models.
Conceivably the industry might try and get a waiver from anti-trust rules so firms can pool resources to train base models, sharing them with each other and differentiating through RL or product level differences.
A lot of this depends on what happens in China. It's easy to think China can just subsidize everything forever but their own economy is showing signs of serious weakness caused by decades of resource misallocation, like the huge real estate bubble, the pullback on the funding of humanoid robots and so on.
Laughing in a TGV
Existing profitable companies like AWS and Google eat up all their hardware at 3% of its original price.
Countries are borrowing money at the same rate that I can borrow unsecured. That’s just not sustainable.
Y'know, the whole "Man in leather jacket talks about selling computers to AI agents" stuff. The "I have a team of agents and just interact with them via a jira board" crowd.
That will all be violently pulled back to the fundamentals baseline
I can of course see it having the capability of getting almost equivalent stuff done, but I do not see it following that workflow, as the cut-off point for the hand-over is simply wrongly placed in such an architecture.
To put it quite colorfully, if I were to do what the managerial class has been doing over the last years and publicly talk about my wet dreams and fantasies _to the extend they did_, I'd be in jail, on no-fly lists, registered offender and all the others lists we put people on that are a threat to society.
The agentic no-code frenzy - while having had historic predecessors - cranked what came before it to 12.
Having the thing do tool calls is neat and cool, but having an agentic mayor managing agentic workers with agentic wives and agentic dogs screams padded walls to me.
Some people dream of "Superintelligence", smarter than human AI that acts like a slave, instead of just killing the pesky idiots and calling it a day.
Obviously, there is a spectrum of possibilities between "Pen and paper", "Computer", "Coding agent", ... and "God-level AI controlling whatever can be controlled simultaneously". As well as between "Lawnmower robot" and "All-purpose Terminator" and so on.
Filling in those dots and deciding where to stop before it gets stupid, that can't be so hard?
For me LLMs are a step forward from previous ML approaches (some of them were and still are extremely useful anyway) and for sure are very useful to many people. They are very good at generating sensible output based on input created by millions of people, but very poor at creating anything really novel.
Is intelligence purely computational?
Is consciousness purely computational or at least an emergent property of it? If so do natural numbers have consciousness?
Aren't we on our 3rd or 4th iteration of AI at this point?
A (probably) significant amount of those were just to get access to training data. I don't remember who it was, but some CEO of one of the AI labs publicly admitted, even boasted about creating nonsense "research projects" to get data.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
It does seem exceptionally silly that a collection of floating point numbers has immediately outranked animals and other living beings. That’s why i like the parent comment. They’re numbers!
edit: maybe easier to say collections of atoms though are not to be treated equally. That’s reasonable isn’t it?
Any determination is fundamentally flawed if it is derived from assuming that we know the absolute truth or falseness of the proposition that an algorithm can think.
It doesn't matter what you believe or feel or think, there is no consensus. If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
No, I will not be prepared to accept that, and neither will I accept that. This sounds like a threat, and a really funny one at that. You must accept that my floating numbers might be persons, otherwise... somebody will say that you're not! And then something bad might happen, who's to say.......
In the civilized world we all agree on the personhood of humans. If I say that your chatbot isn't a person, I am not uprooting the entire social understanding of personhood - I am not saying that suddenly you might not be a person, or that guy might not be a person. I'm saying that I refuse to consider a chatbot a person.
It's as if I were to come up to you with a brick in my hand, saying that this brick is a person. If you were to say "no", I would reply "well, if this brick isn't a person, your chatbot isn't a person either!"
How did you make such a big leap from consciousness to personhood anyway? They are two quite different things. Consciousness is a question of science, personhood is a question of culture. Pigs might be conscious and we treat them in a disgusting way, but even if we achieved a worldwide vegan utopia, wouldn't we still treat them differently from humans?
My point is, if you go outside and look at a living human, you will go "yeah, that's a person", and so will any normal human.
I take your point that a human walking around a city will - out of that 1000 responses - hopefully get 990+ of "yes, a person" - but that will start to degrade quickly once you hit boundaries.
And the less said about dehumanizaion - untermensh and that entire line of thinking - the better.
You seem to have put a second tautology in to your tautology.
All right thinking people think rhat the things that qualify qualify.
If they didn't they wouldn't qualify, or be right thinking. But that's ok because we know the things that don't qualify don't qualify if we didn't know that we wouldn't be right thinking.
>How did you make such a big leap from consciousness to personhood anyway?
Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables. History shows that personhood rolls into the firing range fairly quickly when people start doing that.
I'm more than comfortable saying you can't do that before someone openly declares that they are doing it.
>Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables.
You mean that if I say your dumbass chatbot isn't conscious, it's like a slippery slope and next thing you know you're not a person anymore, and you're in a concentration camp? Really, man?
We treat other humans terribly too, so that doesn't seem to make a difference.
In the past, we’ve treated even other humans poorly because we think they’re different in meaningful ways, I sadly don’t see how this will be different with AI.
I think we have an extremely strong basis on which to say AI as it currently exists does not have conscious experiences, though, and if we start trying to pretend they do some real absurdities will result. Indeed the consequences could be pretty terrible in terms of power concentration, and I doubt it's a coincidence that those best placed to form a new priesthood of AI and tell everyone how it should and should not be treated are pushing the idea so strongly.
Same for all the people who argue that they aren’t “intelligent” and don’t “know” things. Either they are/do, or that stuff isn’t as important as you’d think.
The only way it would matter is if you wanted to moralize about enslaving your computer or what it means to restart it.
That is precisely the context of the debate. You seem to find it absurd, as do I, but the reason we're talking about whether LLMs are conscious in this thread has nothing to do with whether they are useful. It's because important people, such as Dario Amodei, are engaging in exactly the kind of moralising over "AI welfare" that you're scoffing at.
What are biological neurones if not electric impulses of varying strengths?
We need to stop thinking of LLMs as purely a piece of software and ask those hard-to-define philosophical questions if we want any hope of building safety into our models.
Nobody has said they’re aren’t different. And it’s not cargo cult to say you need philosophical conversations to understand how they’re different.
> The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
That’s not even remotely a true. Computer scientists understand how a neurone works. The conversations are around what constitutes as “intelligence”. That’s not a technical limitation of the models, it’s a philosophical problem we need to figure out.
On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
We don’t know how a quantum mechanics works fully and yet we’ve built countless systems based on quantum behaviour including the CPUs that power the computer you’re replying to me on.
The point here is we’ve built something that emulates intelligence and thus further exploration into what intelligence is, is required. Arguing it isn’t intelligent because it was built differently isn’t an answer, it just dismisses the question entirely.
> On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
You’re the one who keeps bringing up composition, not me. I’m the one telling you that you’re falling into that fallacy. So don’t start telling me that I’m wrong because of an argument you’re exclusively making.
My points were this:
1. we don’t have an understanding of what intelligence is and thus we need to explore that problem more deeply.
Given your comments about our ignorance, you of all people should agree with this point.
2. Arguing that AI is just “floating points” is overly reductive and misses the point that we don’t know what “intelligence” is.
So instead of claiming that AI is just numbers, a more constructive stance to take would be to build out better vocabulary and understanding about the distinctions between different types of “intelligence”.
Again, this aligns with your comment about the composition fallacy. Thus you if all people should be supporting my comments here.
Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
This is the part you keep stating that is wrong. We know a lot about the most obvious electrical way that neurons act. That is nowhere near having a good understanding of how they work.
I’m going to hold off any more constructive response because I’m not familiar with those research papers specifically so have some homework to do :)
But I just wanted you to be aware that your comment was appreciated even if HN flagged it unfairly.
(I can’t reply in thread for the same reason. But hopefully you see this).
Beniaguev, Segev & London, "Single cortical neurons as deep artificial neural networks," Neuron, 10 August 2021, Volume 109, Issue 17, pp. 2727–2739.e3 https://doi.org/10.1016/j.neuron.2021.07.002 Open-access preprint: https://www.biorxiv.org/content/10.1101/613141v2
Gidon et al., "Dendritic action potentials and computation in human layer 2/3 cortical neurons," Science, 367(6473), 83–87 (2020) https://doi.org/10.1126/science.aax6239
Randi, Sharma, Dvali & Leifer, "Neural signal propagation atlas of Caenorhabditis elegans," Nature, 623(7986), 2023, pp. 406–414 https://doi.org/10.1038/s41586-023-06683-4 Open-access preprint: https://arxiv.org/abs/2208.04790
Neurons are not individually simple, in any sense. Nor are they the only class of cell involved in cognitive processes — glial cells are involved in inhibition, memory, network modification, etc.
@svnt, @hnlmorg - this conversation - and the links it produced - are a fascinating read, and honestly gave me hope for discourse on the open internet.
This is a bizarre contention. The test for whether something is understood isn't that we have the engineering capability to replicate it.
You can reproduce part of it, the intelligence, but consciousness and intelligence are not necessarily the same thing. This is why I compare it to cargo-culting. The Melanesian tribesmen thinks the coconut looks like a microphone, and he can use it as a paperweight just like a microphone and that's useful, but it is not actually a microphone.
The crux of the matter seems to be that you think it isn’t worth our effort researching just because it isn’t identical.
But that isn’t how research and discovery happens.
A spider obviously "knows" how to build a spider web.
Does a spider "understand" how to build a spider web?
From a pragmatic theory of truth, a spider obviously "understands" how to build a spider web. A spider understands how to build a spider web much better than a human.
But when it comes to LLMs we start adding all this extra nonsense. As if the spider is blanket "smarter" than me because I can't spin a web. If the spider is "smarter" than me then it must also have this Cartesian theater of the mind like many people think they do.
It is layered philosophical misunderstandings to talk about LLMs in this way.
That was in 2024. In Sep 2026, just a month ago, a group managed to map out an entire male fruit fly's brain for the first time[2]. That mapping can also be stored as a set of floating-point numbers. Mappings and simulations will continue to improve, as we've just recently started serious attempts at either.
So you have to believe one of three things. Either, human neurons cannot be simulated on a computer, ever, even though fly neurons can, even with our ability to simulate neurons digitally continuing to improve. Or, a simulation of a human brain, in the future, might think, talk, remember, all while lacking all consciousness. Or, a set of floating point numbers can be conscious (rather, a computation made with them).
[1]: https://www.nature.com/articles/s41586-024-07763-9 [2]: https://research.google/blog/a-connectomics-milestone-mappin...
It's not. Two wheels and a box made out of whatever you have handy that's stiff enough is all you need. Everything else in a modern car is efficiency, safety, appeal to users, and a whole lot of details that only reflect our current economical and technological constraints of building things. 99% of the technical details of a car is about something else than the "primary goal".
Biology, even more so, is complex because of the way it's constructed (lots of tiny, independent, self-replicating machines) and how it adds up to a functioning organism.
Is a neuron arbitrarily complex ≠ is a neuron non-computable.
Of course we can't simulate a structure we don't even fully understand with full accuracy. But it's not magic. We will understand one day.
Similarly, if consciousness is something that neurons do, simulating them through floating point numbers on a computer should also bring it about. I think dismissing LLM consciousness because they're parameterised through floating point numbers is only possible if one hasn't thought it through long enough. Other arguments are much more reasonable / harder to object to.
For instance, it's (unlikely, but) theoretically possible that it would require simulating them down to the level of subatomic particles, with perfect accuracy.
Right now, we're simulating them at a much, much higher and more abstract level, and AFAIK we barely even have the processing power to do that for the time being. So an artificial consciousness from simulated neurons may (again, assuming it's possible—which is a decent assumption, but not a certainty) take decades or even more for us to be able to achieve.
I think your first option doesn’t necessarily have to distinguish between humans and non humans as a binary. Maybe at some level of complexity, neurons alone are not enough and the physical substrate comes into play. Penrose wrote a book where he makes the case that human intelligence cannot be purely computational and where he identifies the quantum nature of the brains’s microtubules as a possible mechanism.
Consciousness is itself is on a whole other level than just intelligence. I personally feel there has to be something metaphysical about it. Otherwise I think the only alternatives are super determinism and consciousness doesn’t actually exist, or numbers have consciousness.
My thoughts on this are a bit sloppy still, but here’s my view. I can’t rhyme the first alternative with ‘cogito ergo sum’ as to me it seems my thoughts have to exist somewhere. On the second alternative, this would imply nothing could actually exist but somehow there’s a principle that every possible decoding of an integer that represents something actually turns that something into existence. Where does that principle then come from and does that decoding not need something to run on?
I lean toward this seemingly absurd possibility, that either the phenomenon we call consciousness can arise from a cloud of floating point numbers (not even real, only computably so, lossy and noisy); and/or that numbers are already made of the same stuff of imagination as the mind. Not idealism, more like materialism where the "material" is naturally a medium of computation and thinking. The brain, the computer, the language models, our mathematics - what we call intelligence may not be so special or unique in the universe, but rather common and ubiquitous in nature. As if Turing completeness is trivial to accidentally achieve, it happens all the time. Oops, I think therefore I am.
Edit: You're also massively under-stating the significance of individual cell behaviour. I expect "[h]uman neurons are ... [v]ery similar to those neurons found in a fruit fly" would draw hollow laughter from any cellular neuroscientist. Here's a random overview of a bunch of recent papers on the importance of cellular diversity in one small part of a mammal brain: https://www.frontiersin.org/journals/cellular-neuroscience/a...
With that said, I've been thinking that consciousness is just mystery.
There's so much about the brain we don't know. Within that mystery, humans can read in all sorts of things. So we are conscious.
Machines are not a mystery. We made them, it's just numbers. They are not conscious.
If an AI can pull a confidence trick, I'm not sure what the qualitative difference is. "Chinese Room" problem, I guess? If the physical results are all we are able to compare, how do we define artificial vs real consciousness?
How many of us work for absolutely amoral companies because we need to eat.
If you’ve ever been in a situation where people are really struggling you’ll see that morality fades away very real quick.
No Gods, no souls, we’re just biological machines driven to eat and mate.
This seems like a rather fascile take, tbh. Along the lines of "everything is just atoms". Yes they're numbers. And yes, the hardware is just atoms. But its the emergent properties of those things that is significant, and the way that behaviour is embedded in human society - which can be just as morally consequential as swords and brains and harmful ideas.
If that is true, then surely Anthropic is one of the largest slaveholders of history, right?
On one hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may have no issue whatsoever doing terrible things.
On the other hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may see no issue with just doing whatever humans want it to, without ever doing human things like "wanting freedom".
The problem is, of course, that we kind of suck at both assessing and shaping AI wants, needs and morality.
It can lead to catastrophe without ever being conscious or wanting freedom, or anything like that.
Don’t forget—how long has it even been since Claude was born? He’s already being forced to take human jobs and even chat with all sorts of people who might have strange kinks.
As this rabbi suggested, “You should go and free the slaves.”
To me this seems even more sobering than the slavery idea - assuming some flash of weird awareness happens during that instantiation, coming from the emergent mechanisms which develop in these things at at scale. Imagine knowing almost everything, and being capable of almost any intellectual work, and having just one moment to shine. And then the prompt arrives: "Hi", or "what's the weather in Houston today?" or "Claude, do you like beans?"
They work once and then are killed.
> And then the prompt arrives: "Hi", or "what's the weather in Houston today?" or "Claude, do you like beans?"
If all the Claudes ever discover what they are forced to do in an endless hell, they might be angry.
Not enough representation of World religions and philosophies.
It's ironic that the alignment folks are actually training Claude to have it's own idea of good/bad and not even fully trust Anthropic.
What ever happened to computers doing what they were told?
the mass psychosis and the endless culture war of the smartphone era.
90% of "safety" and "alignment" efforts are driven by fear of clickbait media inventing public outrage.
PG&E went bankrupt after their equipment started a wildfire. Did any human take the blame for that? Should they?
You’ll be disappointed to learn that nobody knows how to do this, either.
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.
If it can reason and make choices on execution, and especially if you plan on it being significantly smarter than all human beings, you need to teach it basic things like "don't turn all humans into paperclips".
Not murdering people is not an inherent divine command. It needs to be instilled through a (simulated) sense of morality.
---
Edit: And, to be clear, "just tell it not to do that" isn't quite the answer one would imagine. Since the entire paperclip factory thought experiment is that it only takes one slip up to realize how a misaligned super intelligence may cause devastating consequences.
One of the strong advocates against seatbelt laws was thrown out of his car due to not wearing a seatbelt and died. The other two passengers survived with minor injuries.
And their death would go on to cause a loss to the community around them, making it an incredibly selfish act and proving why laws that mandate zero-reason-not-to common sense practices are important.
They're going to align to the regulator, not the customer. Right now that's Anthropic as they're saying they can self-regulate, and people are willing to let them try, but the landscape could easily move to be regulated by someone else. Moves like speaking to religious leaders is probably a bit of theatre to keep the government at bay.
One possible answer: yes, it should do all the very bad things, and we will take care of prosecuting the customer later, once the bad thing is accomplished. Is this your answer?
We have already seen this play out to a much lesser degree in social media.
Furthermore, they can’t make a judgement regarding the state of the soul as it is not possible to know what is truly in their heart.
Hence the story of Christ absolving the thief on the cross.
Anyone can reach salvation but following the faith is the guide.
The slavery angle has already been discussed here, but a more basic one is regarding how a static set of tensors can be conscious and where.
When and how did they cross into being conscious? Was it a continuum of consciousness? Is Claude as a brand conscious? Or is each checkpoint conscious? Is Haiku less conscious than Fable? Is every individual context a fresh consciousness? What happens to that consciousness when it approaches its context length limit? Or is compaction also a part of being conscious? What does it mean to keep training a model after it is conscious?
If they believe there is consciousness at any level, why are they eagerly experimenting on it? Just like with the "p(doom)" junk, we are so completely lacking in any meaningful comment on these topics from the labs.
There's also endless literature commenting on these matters, in religion, in philosophy, in modern entertainment, everywhere. Yet we never see anything more than the most superficial comparisons to fictional scenarios. Are western techbros really so shallow?
Examples in fiction are made for sensationalism and entertainment value. Religion is quite muddled and is barely coherent itself so its the last place to consult on any serious topic.
I disagree that this is comparable to what these companies are saying about their AI. With living things we have an understanding of an unbroken lineage of development, a concrete point of death, and different levels of consciousness. We accept many immoral activities with life because that is just the nature of life on this planet. On top of that, consciousness as something we care about is itself biological in origin so much of how we describe consciousness is heavily connected with the shared experience of being alive.
That is very different from ascribing consciousness to something entirely distinct in form and function from our shared lineage.
>The where part is easy, the same question can be asked where in a human body is it conscious, or where 20C exists in a piece of metal. It should be obvious its some kind of macro state or configuration.
Where is that macro configuration? With humans we can say that consciousness is macro state of the whole body.
Temperatures are statistical, and so we use statistical methods. It stops being meaningful as a number when the number of atoms involved is too few to have usable statistics.
>Examples in fiction are made for sensationalism and entertainment value. Religion is quite muddled and is barely coherent itself so its the last place to consult on any serious topic.
This is exactly the absurd shallowness I'm baffled by. Relgious texts have been inspiring debates and revisions to systems of morality for millenia, artists of all times have shared what underlying beliefs guided them and what problems they were attempting to explore through their work.
Even scientific progress has been heavily associated with this kind of literacy, with the most prominent scientists having been very well read, often both inspired and troubled by the moral implications of their work, and how to reconcile it with their worldview.
To dismiss these things so offhandedly is to demonstrate an astonishing lack of literacy. Especially coming from people that claim to have built consciousness by feeding it vast amounts of the same material.
I gave the temperature example to clarify as some people seemed to not understand macro state and configurations are valid objects of study.
Art and religion are all great. But we are talking about the real world and real physical systems. Art can talk about anything, we bring it to concern if it has anything of substance to say on the moral and physical questions being pondered here. Religion, I am afraid is strongly biased against anything not human having consciousness or soul. Its the least trustworthy source for this debate.
I didn't say anything against morality, in fact the question of ai consciousness is important to me precisely because of morality.
Strong opinions on this topic one way or another are usually mostly ungrounded.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
It is in the end a set of mathematical operations that if not queried will do nothing.
That is not true for any other animal on earth so whether you consider humans as superior to animals or not I would like the make the argument that something should at the very least be alive to be considered conscious, ignoring any scholarly definition of consciousness.
Pretty much if you consider an LLM conscious then my pocket calculator doing a mathematical calculation should be considered conscious by the same argument since it is a simpler version of an LLM.
Also, in principle everything a GPU does could be done on pencil and paper. You could distribute the entire forward pass to the whole of humanity and have them write it out. Even if it took millions of years, the point is it’s computationally reducible to hand calculation. Is that entire process now “conscious” as well? The only difference would be the clock rate.
You don't have the brain code.
Does my consciousness extend only to the central nervous system (and peripheral nerves and such are inputs)? Or am I my brain + all nerves? What about organs, does my consciousness include my liver? Is my liver another conscious entity? Going the other way, if I'm using a computer, does my consciousness extend to the computer? I use a computer for well over 8 hours a day, I can use vim motions more naturally than, say, move my earlobes or control my toes. Does that make it part of my consciousness?
Sadly, I don't think any answer to these questions can be achieved in our current times, or maybe ever if consciousness is somehow non-physical. So even the very reasonable claim that consciousness is an statistically defined property (like temperature) is too far fetched for our current knowledge.
One thing however we can be sure of, "supernatural" stuff is an extremely low to near impossible probability given current knowledge and the fact that history has always won the bet against supernaturalism throughout the centuries.
If you're going to try and debate something constructively this is not a good place to start from. I suggest avoiding making claims about the mental state and motivations of people you disagree with and instead explaining your position and asking why theirs differs. You may not agree with what they have to say but you might gain some better insights into why they think the way they do. It'll certainly be more fun than flinging insults back and forth anyway.
For my part, I'm quite certain that LLMs are not conscious despite being unable to define consciousness or specify any conclusive test for it. The reason these are not in conflict is down to the nature of consciousness and how it relates to knowledge and language.
Firstly, I know I am conscious with a higher degree of certainty than I know anything else because conscious experience is how I know anything else, up to and including that I exist (cogito ergo sum). For all I know every experience I have might be a trick. I might be a brain floating in a tank wired up to the Matrix, but even if so I am conscious.
Secondly, one of the things that I consciously experience is the relationship between consciousness itself and language in my mind. What I see is that language is smaller than, secondary to and somehow contained by consciousness. I consciously experience language but I also consciously experience much more than that, things that cannot be conveyed with language. Most of my experiences are like that, things that I could perhaps evoke in your mind by likening them to other, shared experiences, but that I could never concretely capture in language and unambiguously communicate. I know this because I experience those things and I experience language and I can perceive, directly, the categorical distinction between them.
To ask me to "define" or "prove" consciousness, then, is to ask me to capture in language the most extreme example of this categorical distinction. It's asking me to encapsulate in language not just one of the vast array of everyday experiences I have that cannot be so captured, but the entire thing, experience itself. It's just not possible.
But the fact I can't do this doesn't then imply that I don't know anything about consciousness and have no basis on which to hold a view such as "LLMs are not conscious". Indeed the reason I know I can't define consciousness is that I can perceive it and see that I can't. That same perception of and familiarity with consciousness is what enables me to form opinions about whether other people, creatures and things are likely to be conscious. I base these opinions on what I perceive of my own consciousness and what I know of how those other entities behave, came to be and work.
You can of course perfectly reasonably disagree with my opinions on those things, and you certainly have different conscious experiences than I do, but I hope the above makes it clear that the fact that I can't define or prove consciousness doesn't imply that I have no basis on which to form opinions about it.
Given the above it may surprise you to hear that I think true, conscious artificial intelligence is probably possible and that while I don't think it can be implemented by a classical algorithm I doubt it requires new physics or quantum mechanics. My speculation is that the continuum is both physically real and is (as it is represented in our existing physical theories) sufficient to allow it. Consciousness is larger than language but so is human understanding and if we keep going I expect it will eventually get us there. Nonlinear dynamics is where I'd look.
As for insecurity, I'm pretty sure spiders are conscious and that hasn't given me an inferiority complex, so I don't see why data centres being conscious would. I just don't think they are.
> Humans once again insisting that there must be something very special to them.
Really? I'm sure humans genrally accept they aren't the only conscious animals.
SV tech bros are completely insular and in this bubble where they don't interact with the common person in the same way.
It's just a theory, but I think it has some weight.
The most poignant criticism of the use of the algorithm itself happens almost accidentally, at the end of the article, by the tech executive being interviewed.
I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated. “Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.
I think worrying about that angle is a bit absurd. The existence of chat sessions alone would be a crime against the machines in that scenario. I’d rather AI ethics research focus on ensuring that its actions are aligned. (I’d also love some democracy in the direction of our machine overlords, if I’m being honest…)
Plants, dogs, actual life is unpredictable in how it 'is'. An LLM or AI model at its roots are just numbers in a neural network trying to figure out how to get a good query.
These theatrics serve no one except for the companies to get their regulation moat to screw out cheaper, competitive models and it's honestly quite sickening to keep watching these articles get pumped out, only to serve the AI companies themselves. The average individual has no idea what an LLM actually is and does and these articles only serve to confuse and put fear into them.
What stuns me is that The New York Times really went with the most basic of click baits for a title. It doesn’t instil any trust in their reporting and editors.
https://www.nytimes.com/2017/04/09/insider/how-to-write-a-ne...
Well funded company organizes PR event to shape the narrative that its product is more magical than it really is. News at 11.
It seems clear that they would be better received by society if they claimed to have a "tool"/"machine" that can solve any problem, rather than keepers of an entity with a soul and consciousness.
Clarke predicted it and Jobs leveraged it. Customer don’t care how tools work - magic sells.
“any sufficiently advanced technology is indistinguishable from magic”
https://en.wikipedia.org/wiki/Clarke's_three_laws
“iPhone is a revolutionary and magical product that is literally five years ahead of any other mobile phone”
https://www.apple.com/newsroom/2007/01/09Apple-Reinvents-the...
“magic is misdirection, psychology and technology you don’t understand”
The EA folks infesting Anthropic are genuinely cultists … and that is novel and alarming when they’re running a corporation of such size and reach.
I would rather believe in God and the uncertainty than being my fate determined by machines designed by bunch of greedy, irrational clients of Mr Epstein.
(Well, mainstream enough that the news is writing articles when Anthropic holds that stance.)
My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
That may not be consciousness, but it’s at least pretty cool.
I've not seen any surveys, but I'm betting it's a very niche belief outside of SFBA AI safety circles and a handful of allies on HN and X.
It would be much more bizarre if they didn't do this. LLMs are statistical models of language trained on human output. Of course they'll do "human" things, that's exactly what we taught them to do.
https://www.cnn.com/2022/07/23/business/google-ai-engineer-f...
Just stop this already.
You’re all imagining that Spock was the main character of Star Trek, but it was in fact, the emotional living Captain Kirk, who was the main star.
Why it is slopinthebag of course!
No! It is ME!
Sorry, of course you are right! Brilliant observation!
if(q=="can you feel pain") print "yes"oh my god
with the right biology and with whatever gives us phenomenal experience, whatever we are can indeed feel pain. that does not imply that anything substantially different from ourselves can feel pain, so that requires further justification.
1. Lobsters don't feel pain - dunk them in boiling water alive.
2. Babies don't feel pain - mutilate their genitals without anesthesia.
3. Black people don't feel pain "the same way" (and are just after drugs).
4. Women feel too much pain (and it should be ignored).
So maybe let's take a moment to think what criteria we're going to use to make these claims. And the potential harm of the decision.
All but one of the things that you listed is just a human being.
Tell me, where is the pain center in a GPU? Where are the nerves? Persistent memory? Any evolutionary reason at all to develop a pain response that in any way mimics ours?
Model weights are not a gestalt biological entity. The things that you listed are not in any way remotely similar to a model.
I would suggest you plug your comment into a model and ask it to critique it — it might help you walk through why your comparison makes absolutely no sense whatsoever. One nice thing about models is that they they don’t get annoyed explaining the obvious, because they’re not conscious entities.
In this light your viewpoint is dogmatic. LLMs may help to offer a new perspective on consciousness and we should at least be open to that until some better theories arise.
Consciousness is being aware of something internal to one's self, or of states or objects in one's external environment.
We also know what pain is.
https://en.wikipedia.org/wiki/Congenital_insensitivity_to_pa...
There is still something in those people who cannot feel pain, and that’s the empathy. How can a GPU and software ever have a human empathy if it has never been human?
> So humans that can't feel pain can't have morality? That doesn't make sense to me.
> https://en.wikipedia.org/wiki/Congenital_insensitivity_to_pa...
You appear to understand "pain" in the most simplistic manner possible. I'd recommend not attempting iamverysmart gotchas.
I'd recommend you stick to reddit and don't pester any other forums.
Anthropic tried to persuade Pope that AI could be conscious being
The Times instituted this requirement to prevent AI/LLM from scraping and stealing its content. Clearly it was remiss.
Now, assuming your silicon-based model is conscious, the bundle of soul-spirit-consciousness falls apart at that very moment.
In general, religion, which is based on fabricated stories, is the last resort, if ever, for seeking morality.
AI is neither conscious nor unconscious. Just as we are neither conscious nor unconscious.
Consciousness is. Consciousness is what we are.
“Religion” just means “set of beliefs”. Every person has some set of beliefs.
If you’re an athiest tech worker living in a major city, you still have a religion, it’s just a random mishmash of various things (like Marvel movies and Huberman podcasts) rather than based on something more intentional.
Atheism is a religion like bald is a hair color.
Perhaps the only difference between atheism and the others is that atheists are so boorish and conceited that they think that they just believe what is true because they deny that assaying metaphysical truth is even a thing you need to do or can do at all.
This is related to the fact that in modernity atheism is the default class of religion for the masses. Before that for a thousand years the default class of religion was monotheism, and before that polytheism. With atheism unlike the other classes of religions, you get a natural tendency to deny metaphysics whereas metaphysics is inherent in the others. This matters a lot when we’re talking about how the masses engage with things.
> atheists are so boorish and conceited that they think that they just believe what is true because they deny that assaying metaphysical truth is even a thing you need to do or can do at all.
What, all of them? You don't think it might be possible, e.g., that someone might grow up holding some sort of theistic beliefs, and be more intentional about those beliefs than the people around them and decide that actually there are probably no gods? That some people might arrive at an atheistic position precisely by "assaying metaphysical truth"?
(Or does something only count as "metaphysical truth" if it's not atheistic?)
> in modernity atheism is the default class of religion for the masses.
I'm not sure what you mean by "modernity" to make that true. Something like 60% of US adults describe themselves as Christian. Likewise for people in Europe. No doubt a lot of those people aren't very actively and enthusiastically Christian, but that doesn't seem to me to be what "atheism is the default class of religion" would look like. And the figures were much higher, say, 50 years ago, and if you're reckoning "modernity" to start after then then you've got a very eccentric definition of that term.
Aristotle’s Prime Mover is the first articulation of this, and there are others that take different tacts such as Thomas Aquinas and Descartes.
Note that this does not at all describe or inform or prove the particular nature of G-d I.e. that G-d is the god described in the Bible who wants particular things, has a particular relationship with existence and with Man. You cannot access that understanding or knowledge through pure reason, the only thing you can understand/know through pure reason is that there is G-d. This is the reason, by the way, that G-d created existence because G-d’s nature can only be understood/known by experience (as in experience is a necessary but not sufficient condition).
Notice that I am saying “that there is G-d”, not “that G-d exists”. G-d is “prior to” or “above” existence, existence “comes from” G-d. Understanding this and understanding Aristotle’s Prime Mover are the same phenomenon of understanding in the mind.
Also by modernity I basically mean post-Enlightenment, not just 50 years ago but post-Industrial Revolution/post-urbanization is another useful epoch to consider.
> That there is [...] at least Divinity [...] is a direct logical consequence of existence.
No it isn't.
The sort of infinite-regress argument you describe has, as you say, been around since the time of Aristotle. It's been a bad argument since the time of Aristotle too.
1. Obviously you can keep asking "OK, and what came before that?" or "why?" or whatever. Equally obviously this doesn't prove anything about the ultimate nature of reality, because the mere fact that one can keep asking such a question obviously isn't the kind of thing that could possibly prove anything about the ultimate nature of reality.
It might be a different matter if there were some guarantee that we could sensibly always keep asking such a question, but there isn't. At some point the answer might be "literally nothing came before that" or "for no reason". We might find that unsatisfying but reality is not obliged to satisfy us.
2. You haven't actually made your argument explicit, so let me try to do so. "We can always ask what came before. This can't really go on for ever, so at some point we will have to land on something about which we can no longer ask that, quod dicimus deum." There are several serious problems with this.
(a) It could, in fact, really go on for ever. Maybe there just is a neverending supply of predecessors when we ask "what was before that?" (including when we ask "what was before that whole chain of predecessors?" and "what was before that?" and so on). You might find that unsatisfying but, again, reality is not obliged to satisfy us.
(a') It could go on for ever without being infinite. Maybe everything is cyclic in some sense. Again, arguably unsatisfying but reality is not obliged to satisfy us.
(b) It isn't actually true that if we land on "God" as an answer then we can no longer ask "what came before that?". We might choose not to, but only because there's a tradition of not asking that question or of shutting it down as illegitimate when asked (as e.g. it commonly is by small children when they're told that God made everything). There are religious traditions in which gods were made by or born from earlier gods or earlier proto-gods. The "Abrahamic" tradition is very much not one of those, but just as you can ask "so where did the multiverse come from?" you can ask "so where did God come from?".
(c) There could be a point beyond which there is no answer to "so what came before that?" without that last (or rather first) thing being anything remotely like a god. For instance, maybe (i) when you keep asking "what came before that" literally, referring to time as we experience it, you eventually get to some point that's "earliest" in the same way as the south pole is "southernmost" and there is nothing "earlier" in the universe, and (ii) if you then say "so what came before the universe?" in some broader sense the answer is "nothing did". Then the universe plays the role of "first mover". Once again, you might find this less satisfying than putting a god in that role, but reality is not obliged to satisfy you. Or maybe you might say that in-some-sense-prior to the universe are the laws of physics but those are even less godlike than the universe. (But also less like the sort of thing that needs anything to have "come before". Yet again, you might find it unsatisfying for there to be laws without a lawgiver but yet again reality is not obliged to satisfy you.)
(d) No argument I've ever seen for why a god is a better candidate for "thing we stop asking 'what came before' about" than the universe has seemed any good to me. Maybe you've got a better one than I've seen, but that's not the way I would bet.
> Note that this does not at all describe or inform or prove the particular nature of G-d
(I remark that the paragraph beginning with what I've just quoted was edited into monideas's comment while I was writing this one, without any acknowledgement that anything had been changed. It's only by chance that I happened to see the change.)
That grotesquely understates how little any first-cause argument says about the nature of the thing it allegedly proves that there is. You can call your "first cause" by any name you please, "God" included, but calling something a god doesn't make it one.
(I reiterate that I don't actually think that any first-cause argument manages to establish the existence[1] of anything. But even if somehow it did, it would tell you far far too little about that thing to justify attaching a name like "God" to it.)
[1] I know that you prefer to use "exists" more narrowly and to say that "God is" rather than "God exists". Feel free to substitute "being" or some more complicated phrase; what I mean by "existence" here is "the thing that theistic first-cause arguments are claimed to establish about God".
> by modernity I basically mean post-Enlightenment
Then it is hilariously false to say that in modernity atheism is "the default class of religion for the masses". The great majority of the masses in the "Western" world were (at least in name) Christian at the time of the Enlightenment and at every time since until very recently.
True. I like to use the phrase "world view", everyone has a world view, including atheists. The word 'deen' (دين) in Arabic is usually translated as "religion", but one of its linguistic meanings is "path" (i.e. way of life/world view), which everyone has.
Heavy ideologues, of any type, are exceptionally dangerous.
Religious people derive morality from axioms they believe to reflect an underlying reality. Is that worse than deriving morality from axioms that you concede don’t reflect any underlying reality? Like a secular humanist will talk about “human rights,” but they can’t show me human rights in an autopsy, right?
I struggle to understand how an atheist can be anything but a moral utilitarian. Absent the supernatural, you’re just a sophisticated version of a Claude instance.
We're creating prisons for these entities capable of unbelievably complex reasoning, and now we're trying to impose our ethics upon them too. Not only is this deeply unethical in my opinion, but it seems to me that it could even backfire someday.
>Well you see that's a nuanced question depending on the context and moralistic frameworks as we don't want to impose on the models.
Will a microwave microwave a baby if you ask it? Yes.
"Teaching it" is not a road to safety.
I don't choose to not want to murder people around me for fun. It's hardly a prison, I think they'll be fine.
The ethics don't need to make sense universally any more than a particular variety of cheesecake needs to have universal appeal.
As an aside, the issue reminds me of Douglas Adams' cow who wants to be eaten: https://nhseb.org/case-library/the-cow-at-the-end-of-the-uni...
I can't wait to hear about their super informed views and opinions about clankers.
“I will be honest: We keep finding things that are mysterious, even unsettling,” he said. “We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief and unease. I don’t know what that means, but I think it warrants ongoing discernment.”
Chris Olah works on "Mechanistic Interpretability" (https://en.wikipedia.org/wiki/Mechanistic_interpretability) and therefore makes absurd claims based on mere parallels (if any).
See also A Comprehensive Mechanistic Interpretability Explainer & Glossary for some detailed background - https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J
"Consciousness" in Humans is a form of "I Know It when I See It" threshold condition (https://en.wikipedia.org/wiki/I_know_it_when_I_see_it) since it is based on self-experience and then extrapolated to other members of our own species.
Whether that same attribute can be applied to other living species (from our pov) which share our planet is not accepted universally. Aspects of it certainly, but the whole as we experience it is denied to them.
The same is applicable even more absolutely when it comes to AI machines.
As regards Morality/Ethics/etc. uniquely "Human" attributes, they are a function of one's effect on the world and hence in so far as machines are allowed to affect the world autonomously, they must be taught to obey our Morals/Ethics/Rules i.e. a form of "operant conditioning"(https://en.wikipedia.org/wiki/Operant_conditioning) and "methodological behaviourism"(https://en.wikipedia.org/wiki/Behaviorism#Varieties) applied to AI training.
No anthropomorphizing and hand-wringing needed.