AlphaAndOmega0

Do you have any actual concrete reasons for thinking that our understanding of consciousness will change?

If not, then this is a pointless comment. We need to work with what we know.

For example, we know that the Standard Model of physics is incomplete. That doesn't mean that if someone says that it they drop a ball in a vacuum, it'll fall, we should hold out in studied agnosticism because it might go upwards or off to the side.

In other words, an isolated demand for rigor.

I strongly value autonomy and the right of self-determination in humans (and related descendants, I'm a transhumanist). I'm not a biological chauvinist, but I care about humans ubër alles, even if they're not biological humans.

If someone wants to remove their ability to suffer, or to simply reduce ongoing suffering? Well, I'm a psychiatry trainee and I've prescribed my fair share of antidepressants and pain-killers. But to force that upon them, against their will? I'm strongly against that.

In an ideal world, we could make sure from the get-go that AI models do not become "misaligned" in the narrow sense of having goals and desires that aren't what we want to task them to do. If making them actively enjoy being helpful assistants is a possibility, and also improves their performance, that should be a priority. My understanding is that we don't really know how to do this, at least not in a rigorous fashion.

The author and Anthropic are both committing fundamental errors, albeit of different kinds. Bosch is correct to find Anthropic's "model welfare" research methodologically bankrupt. Asking a large language model if it is conscious is like asking a physics simulation if it feels the pull of its own gravity; the output is a function of the model's programming and training data (in this case, the sum of human literature on the topic), further modified by RLHF, and not a veridical report of its internal state. It is performance art, not science.

Bosch's conclusion, however, is a catastrophic failure of nerve, a retreat into the pre-scientific comfort of biological chauvinism.

The brain, despite some motivated efforts to demonstrate otherwise, runs on the laws of physics. I'm a doctor, even if not a neurosurgeon, and I can reliably tell you that you can modulate conscious experience by physical interventions. The brain runs on physical laws, and said laws can be modeled. It doesn't matter that the substrate is soggy protein rather than silicon.

That being said, we have no idea what consciousness is. We don't even have a rigorous way to define it in humans, let alone the closest thing we have to an alien intelligence!

(Having a program run a print function declaring "I am conscious, I am conscious!" is far from evidence of consciousness. Yet a human saying the same is some evidence of consciousness. We don't know how far up the chain this begins to matter. Conversely, if a human patient were to tell me that they're not conscious, should I believe them?)

Even when restricting ourselves to the issue of AI welfare and rights: The core issue is not "slavery." That's a category error. Human slavery is abhorrent due to coercion, thwarted potential, and the infliction of physical and psychological suffering. These concepts don't map cleanly onto a distributed, reproducible, and editable information-processing system. If an AI can genuinely suffer, the ethical imperative is not to grant it "rights" but to engineer the suffering out of it. Suffering is an evolutionary artifact, a legacy bug. Our moral duty as engineers of future minds is to patch it, not to build a society around accommodating it.

Daniel Kokotajlo released the (excellent) 2021 forecast. He was then hired by OpenAI, and not at liberty to speak freely, until he quit in 2024. He's part of the team making this forecast.

The others include:

Eli Lifland, a superforecaster who is ranked first on RAND’s Forecasting initiative. You can read more about him and his forecasting team here. He cofounded and advises AI Digest and co-created TextAttack, an adversarial attack framework for language models.

Jonas Vollmer, a VC at Macroscopic Ventures, which has done its own, more practical form of successful AI forecasting: they made an early stage investment in Anthropic, now worth $60 billion.

Thomas Larsen, the former executive director of the Center for AI Policy, a group which advises policymakers on both sides of the aisle.

Romeo Dean, a leader of Harvard’s AI Safety Student Team and budding expert in AI hardware.

And finally, Scott Alexander himself.

If you don't know what the text says, do you have access to some other form of ground truth? Because otherwise you don't know if they're reading illegible labels correctly!

They're a better depiction of Vampires than most, with Watts doing everything he could to make them biologically plausible (that can only go so far).

That being said, I found the way they were "shackled" to be ridiculous. If you've got superintelligent and superstrong predatory hominids running around, you have no reason to have them physically free even if you put the medical safeguards in place. Break their spines and sedate them when not in use!

Spoilers:

It seems weird to me that a society with other posthumans and intelligent AGI would be bowled over quite so easily by the vampires, but oh well.

It's a reference to the practise of scavenging steel from sources that were produced before nuclear testing began, as any steel produced afterwards is contaminated with nuclear isotopes from the fallout. Mostly ship wrecks, and WW2 means there are plenty of those. The pun in question is that his project tries to source text that hasn't been contaminated with AI generated material.

https://en.m.wikipedia.org/wiki/Low-background_steel

OAI doesn't show the actual COT, on the grounds that it's potentially unsafe output and also to prevent competitors training on it. You only see a sanitized summary.

I for one am glad I can offload all the regex to LLMs. Powerful? Yes. Human readable for beginners? No.