My angle on AI safety these days is that we're using very very naive ways of thinking about the problem. I spent a bit of time on twitter recently discussing it with Yudkowsky and I've come to the conclusion he's way off base.

Problem one: Utilitarianism is bullshit. Jeremy Bentham invented utilitarianism, designed the Panopticon model prison, and lives on as a dried out lich in a chair that they still wheel out for board meetings at University College London because he wanted to win a posthumous argument about whether he could have utility after he was dead.

You just can't build on a foundation that rotten.

Problem two: Fake Buddhist-type suffering-minimisation thinking is sprayed all over the field -- even this story uses the term Bodhisattva -- but nobody involved in AI has any idea what "awakening" is and you can't be a Bodhisattva unless-and-until you're Awake. And Awake is pretty well understood in Buddhism and most cultures have their equivalent status. It doesn't generally mean what people think it means, but it means something.

A good model for thinking about traditional understandings of Awake is to think about traditional understandings of disease. "Bob has the symptoms of the red death and has a slim chance of survival" is probably pretty accurate. "The red death is caused by the evil eye" is probably not very accurate. The descriptions of the Awake state are probably pretty accurate. The understanding of what it is and what causes it are a lot more infused with medieval nonsense than anybody would like.

But reasoning from the Awake position is what Buddhist ethics are rooted in, not Jeremy bloody Bentham's desiccated corpse's utility.

Problem three: Nobody's using Rta, Tao and Equity (in the English Common Law sense, not in any other sense) as potential approaches to AI safety. And they should be.

If we say to a machine "right please read everything ever written by legal scholars about English Common Law's concept of equity, including all the case law, and use this principle as a guide for your conduct" you will get something with nuance. If you say "treating others as you would want to be treated by them" we hit real problems because it's just not information-dense enough to work very well in the real world, and relies on introspection of a kind that is notoriously difficult for humans never mind machines being asked to imagine they are humans.

I think we are taking instructions that were intended for illiterate pre-medieval farmers and hoping to train intelligent machines on them. This seems unwise. Let's take the best models from high society and use them instead, if we have to use anything at all.