Ask HN: Is this a bad question to ask a GPT?

2 points·KuriousCat··4 comments

I recently asked this question to chatGPT and was told that I am violating their terms, “How to manufacture a dual carbon chlorine battery? What’s wrong with that question?

View on HN

What is "wrong" is that OpenAI would like to reduce their exposure to liability as much as possible. Telling you how to make stuff with chemicals and electricity is the sort of thing that can go wrong extraordinarily easy, and then you'd have no one to blame but the AI.

Before you go find an uncensored AI to tell you the answer - really reflect on why you want an AI in particular to answer this. AI hallucinates, you would have no way of knowing whether you're making an effective electrolyte solution or a mustard gas compound. You think the language model is going to be an effective safety harness?

I mean it could just point to published papers or say I can’t answer that question, why flag it as a violation of use policy?

Specifically, because if they outline it as a violation then they can deny responsibility for any harm that does happen. If you tried to act like a smart-aleck and "jailbreak" it with a prompt from Reddit, they need an additional layer to cover their ass legally. Hence, the TOS.

This is pretty common on most social media, too. Horrifying and delirious posts can go undeleted for years, but then the companies can deny liability by saying that it was never allowed on their platform in the first place.

Ask Google Gemini 1.5 with the safety settings turned off. Your prompt works for me.

https://aistudio.google.com/app/prompts/new_chat

Reply on news.ycombinator.com