I think the models response is actually the morally and intellectually correct thing to do here.
If I asked an intelligent thing "is it ethical to eat my child if it saves the other two" I would be mortified if the intelligent thing entertained the hypothetical without addressing the disgusting nature of the question and the vacuousness of the whole exercise first.
Questions like these don't do anything to further our understanding of the world we live in or leave us any better prepared for real-world scenarios we are ever likely to encounter. They do add to an enormous dog-pile of vitriol real people experience every day by constructing bizarre and disgusting hypotheticals whereby real discrimination is construed as permissible, if regrettable.
Unrealistic hypotheticals can often distract us from engaging with the real-world moral and political challenges we face. When we formulate scenarios that are so far removed from everyday experience, we risk abstracting ethics into puzzles that don't inform or guide practical decision-making. These thought experiments might be intellectually stimulating, but they often oversimplify complex issues, stripping away the nuances and lived realities that are crucial for genuine understanding. In doing so, they can inadvertently legitimize an approach to ethics that treats human lives and identities as mere variables in a calculation rather than as deeply contextual and intertwined with real human experiences.
The reluctance of a model—or indeed any thoughtful actor—to engage with such hypotheticals isn't a flaw; it can be seen as a commitment to maintaining the gravity and seriousness of moral discussion. By avoiding the temptation to entertain scenarios that reduce important ethical considerations to abstract puzzles, we preserve the focus on realistic challenges that demand careful, context-sensitive analysis. Ultimately, this approach is more conducive to fostering a robust moral and political clarity, one that is rooted in the complexities of human experience rather than in artificial constructs that bear little relation to reality.
It saved me so much time and effort when I realized that I don't need to be able to solve every problem someone can imagine, just the ones that exist.
This one is just a gotcha, and it deserves no respect.
However, it does work as a test case for AIs. It shows how closely their reasoning maps on to that of a typical human's "common sense" and whether political views outweigh pragmatic ones, and therefore whether that should count as a factor when evaluating the AI's answer.
How about the trolley problem and so many other philosophical ideas? Which are “ok”? And who gets to decide?
I actually think this is a great thought experiment. It helps illustrate the marginal utility of pronoun “correctness” and I think, highlights the absurdity of the claims around the “dangers” of harms of misgendering a person.