Hello, I saw the pingback for this thread today. I'm surprised (and a bit giddy) that my obscure little blog wound up on HN, so I hope you don't mind my coming in a little late.
The purpose of this post was more exploratory, rather than for the sake of drawing specific conclusions, which I am aware can be frustrating. Thanks for taking the time to read it anyway.
You are correct that I was not very interested in formalizing ethics. I'm not so sure why OP chose to use that to title the thread, because only a part of the post was oriented towards overviewing the AI safety discourse, and within that discourse, the development of a complete formal system for ethical reasoning is considered a naive way to attack the problem. I never wrote about the formalization angle directly.
The problem, such as it is, is that it's troublesome to encode what we want of an AI agent, even more so to encode how it should reason about what we want of it under unforeseen circumstances, no matter what it is that we want. Ultimately, in order to successfully implement regulation, an agent must be "taught" (in whatever sense) to make judgments according to what is wanted of it. Technically speaking, this is a hard problem. However, there does not necessarily need to be an absolute answer as to what the correct ethics is, in order to teach a machine methods of reasoning about ethical problems, nor for us to reason about what sorts of imperatives a machine should be reasoning about.
The thrust of that section, however, was to raise an issue with the (in my opinion) narrow way in which that problem is typically formulated, to which you have also alluded. Specifically, my beef was ontological (which, for the sake of communication, was distilled into the bottom-up/top-down distinction), but there are conceivably other gaps.
This could have been elaborated; ontological approach is one of the ways in which ethical traditions are distinguished, but far from the only one. Any distinguishing factor may have been missed by an approach like IAD.
> That's why I think robot ethics is kind of misguided. What we need is laws that regulate AI and put its use into a legal framework and closes loopholes. This is a political, not just an ethical issue.
This much I agree with wholeheartedly. I made the case that we shouldn't shaft the ethics for the politics, but the same goes for the reverse. Where how we consider ethical problems comes into play again in the second case, is that putting the use of AI into a legal framework and "closing loopholes" has roughly the same shape as the problem of AI ethics, it just chooses potentially different parties to determine what it is we want of the agents.
Anyway, thanks again! If you have a moment, I'd be interested in reading what you thought was naive in my approach.