The broader problem is that current AI and AI companies are not aligned with humans, human values, or human flourishing. AI is a powerful tool that can be aimed at quantified goals but we don't actually know how to quantify human values and flourishing (Goodhart's Law).