Right now goal-setting is something intelligences do not, and cannot, do for themselves. Humans must define the bounds of a problem carefully before a robot brain can perform useful work (some kind of numerical optimization).
The preliminary problem, then, is: how do humans define goals?
And the final problem: construct an intelligence that is able to efficiently set and achieve goals that are broadly in line with human goals.
I think this statement of the problems neatly sums up my difficulty with the notion of "strong AI", or "AGI", or Robot God or what-have-you and the possibility that it might be somehow useful in the world.
Because the way humans set goals, I think, is through vague heuristics that are represented as narratives carried by culture and society; we hold these narratives and pass them back and forth to each other, through various tongues and modes of fashion.
This means that human desire is the product of a constantly-shifting stream of socialization, which we are all drinking from and pissing into at once. The only meaningful way to accurately represent this, I think, is for engagement in it. You must participate in culture to "get it". Where this participation breaks down ("let them eat cake") we get strife.
Where does this leave the poor robot mind? It can only be "intelligent" in the way that we want when it can appreciate the horror of losing its daughter to a prison camp, when it can come to feel the memory of an inherited tragedy as both burr and serious weight. At this point we're just raising children again.
At any other point it's simply a dumb slave, doing exactly what we tell it - or a capricious, self-serving monster to be fought.