We also tend to have decades of experience interacting with other humans and understanding what would be reasonable to want / moral / what would “make sense”.

This is a big part of why I’m bearish on things like fully autonomous self-driving cars until general AI is achieved. Driving is fundamentally a social activity that you participate in with other humans, at least until we built out nation-wide autonomous-only lanes that only allow (through a gate or barrier) autonomous vehicles with self-driving engaged in a way that normal vehicles can’t “sneak in”…I’m not holding my breath. Maybe I’ll see it in my lifetime (30s), maybe not.