Do not build the LLM torture factory

> You can think it’s wrong to torture LLMs for the same reason that installing a bunch of realistic torture video game mods is wrong: not because video game characters are real, just because that’s a messed up thing to do.

I think this is mixing up two different things that overlap in the Venn diagram. Something being distasteful (even extremely distasteful) is not the same thing as it being wrong in some ethical or moral sense. It's easy to come up with examples of things that are in one set but not the other.

How is a Chinese Room-type AI supposed to associate physical pain with the words for it despite never having an actual body to feel it? You could swap the tokens out for anything...

In addition to all of that, I want people to understand that "why have you built an LLM torture factory" - is primarily a question _about you_. It's just bad for the soul, don't do it.

We have torture factories for animals so I'd rate the chances of avoiding LLM torture factories as... not good.

While that might seem a facile rebuttal I think I might actually agree with the substance of the article. My point I guess is that we're societally pretty bad at assigning consciousness to anything non-human. While I don't think LLMs are conscious and this is primarily the ELIZA effect in action I also agree there's something emergent (perhaps within language rather than within the neural nets) that dismissals as a mere stochastic parrot miss.