I've been thinking about this problem with law a bit lately. It seems like our laywers & politicians want every legal question to be completely decidable, and this attitude leaks into popular culture too (or is it the other way around?), hence this article's preoccupation with the fact that good or evil intent cannot (as far as we know) be identified by any mechanical process.

This type of thing is of course a problem in computer science, but the law is not a computer, and I wish we could stop treating it like one. With humans involved, it's quite feasible to make decisions about undecidable things. This process is called exercising judgement, and we hire judges to do it. Frankly, I don't see what the problem is with a "don't be evil" clause. Pretty much everyone understands what it means in most any given circumstance, even if we can't write down a decision procedure for it.

I understand that part of the motivation here is the urge to CYA and offload all accountability for bad decisions from people to "the law". Not sure what to do about that.

Further reading: http://www.newyorker.com/arts/critics/atlarge/2012/01/30/120... ; appeared on HN yesterday; from paragraph 7, argues that these attitudes are partly responsible for the current incarceration problem in America.

(I wonder, can we make a proof of the undecidability of the good-vs-evil problem? I don't think it'll be as easy as for the halting problem; we know how to make a program halt or not halt, but it's unclear how to make a program "do something evil" or "do something good".)