Hacker News Reader

We are part of society, so I hope we don't "trained a perfectly rational AI" but train something that operates within constraints that we agree upon as a society.

A "perfectly rational AI" might also kill of disabled people as unproductive, but we as a society will need to introduce bounds and constraints and objectives in line with our human values.

Reasoning that can also justify eugenics or other heinous stuff.

The opposite reasoning justifies "Harrison Bergeron".

https://books.google.co.in/books/about/Weapons_of_Math_Destr...

No, assuming that the goal of your perfectly rational AI is to maximize profits, you can change the decision by adding a disincentive for choosing against women.

Which is what governments usually do.