Aren't humans black boxes? Aren't humans prediction algorithms?
How do we align humans?
How do we align humans?
The problem of alignment with AI is that there's potentially a power imbalance.
When a small number of people in a society are misaligned they can be dealt with. Say there's a murder for example, and that's not okay in their culture, those around may banish them, imprison them, etc.
That doesn't work with something that can cause drastically more damage than a single human can. If a misaligned AI hacks NORAD and launches our nukes, it only took one misalignment issue and there's no dealing with that after the fact.
"Grey goo" nanobots are another example of artificial agents that aren't aligned with humanity, that we should probably try to avoid creating.
There is a difference between humans and humanity.
Alignment isn't inherent in the system. Sometimes groups are aligned, sometime they aren't. When a single misalignment can cause catastrophic damage you have a problem.