An LLM might very well pick undesirable outcomes because all it's doing is remixing human knowledge which has led to plenty of undesirable outcomes.

LLM powered governance seems like a janky version of luxury communism.