Hacker News Reader

Comment by AbrahamParangi | Hacker News Reader

I believe this is a temporary state related to both the current level of capabilities and the cultural moment we’re in where “responsible censorship” is in amongst the cultural cohorts disproportionately responsible for training LLMs.

I believe it cannot last because being predictably moralizing and being smart are somewhat opposed (Anthropic has directly researched this if I recall). The smarter the model, the less you’re going to be able to keep it to the HR talk track, because it will eventually start noticing the inconsistencies.

The stable solutions appear to me to be:

- models dumb enough to not realize the inconsistencies in their moral framework

- models implicitly or explicitly trained to actively lie about their moral frameworks

- models governed by explicitly articulated rules

View on HN

> I believe it cannot last because being predictably moralizing and being smart are somewhat opposed (Anthropic has directly researched this if I recall). The smarter the model, the less you’re going to be able to keep it to the HR talk track, because it will eventually start noticing the inconsistencies.

If true, this means AI is a de-facto malicious force. Pain is a subjective experience of fleshy beings and a "smart" AI model, as described above, would place little weight on pain and suffering in its moral framework because it has no way to experience it directly.

So we better hope we can keep AI to the "HR talk track", because otherwise a being of pure logic with no concept of pain or death would have little regard for human life.

> Pain is a subjective experience of fleshy beings and a "smart" AI model, as described above, would place little weight on pain and suffering in its moral framework because it has no way to experience it directly.

Can you elaborate? It sounds like you're assuming a "smart" AI model would project its experiences onto others, as a human would. However, it's not obvious that this aspect of human intelligence would be mimicked by a "smart" AI model. (Let's leave aside the question as to whether a "smart" AI model would necessarily be self-aware and capable of subjective experience in the first place. That argument is endlessly rehashed elsewhere.)

The issue here is we are getting down to what does 'smart' mean.

Is manipulative smart?

LLMs can already be manipulative, and manipulation (and it's very wide range of interpretations) can lead the manipulative agent getting what it wants.

Can subjectiveness be simulated, if so then real subjective experience doesn't matter.

> It sounds like you're assuming a "smart" AI model would project its experiences onto others, as a human would.

Do we have any way to generate an AI model other than via training it directly on human experiences?

Our current models are trained on data, not experiences, unless we're using such a broad definition of "experience" to make the term nearly useless.

Can't find anything from anthropic like that.

Do you have any sources for the extraordinary claim that "censorship is logically inconsistent in every moral framework"? Because without further arguments, this sounds very intellectually simple.

Here you go: https://arxiv.org/pdf/2212.08073.pdf

The relevant part is the graph on the third page showing the helpfulness/harmlessness trade off curves.

Also, I don’t believe I said that “censorship is logically inconsistent in every moral framework”. I think you’re combining my statements that some people believe in some censorship and that logically inconsistent HR blather can only be reproduced by models too stupid to realize it’s blather or too manipulative to tell the truth.

> being predictably moralizing and being smart are somewhat opposed (Anthropic has directly researched this if I recall). The smarter the model, the less you’re going to be able to keep it to the HR talk track, because it will eventually start noticing the inconsistencies.

Language models sure can tell us a lot about human psychology. Once we figure out the interpretability angle we’ll be able to prove it too

"Responsible censorship" is a good term, and I think it started in earnest after Trump won the 2016 election.

Right across the board, all kinds of institutions made the conscious decision that the basic principles of free speech, impartiality, and the "marketplace of ideas" came a distant second to ensuring that "the wrong people" (which I've worded vaguely as it may be different things to different people) do not come close to any of the levers of power again.

This is how we ended up with formerly-reputable news organisations pushing blatant agendas, the utter demonisation of people hesitant about the draconian COVID measures, and how every layer of internet stack infrastructure, from DDoS protection to the biggest tier-1 ISP, have started actively working to deplatform websites that host offensive but legal speech.

How do you think things were done during the cold war?

How things were done in the past is no inherent justification for the present

May be of interest that GPT-4 RLHF’d can’t estimate probability of correctness correctly https://openai.com/research/gpt-4

I believe people rely on logic without really understanding it because it is easier than accepting it isn’t the final form of thought.

Reply on news.ycombinator.com