With reinforcement learning positive rewards are assigned for positive outcomes (a “carrot”) and negative rewards are assigned for negative outcomes (a “stick”).
The model is trained to maximize the expected rewards per decision. I’d say this is fairly analogous to desire.
I’ve written an introduction to reinforcement learning here: https://improve.ai/reinforcement-learning/
1. This "desire" applies only to the training algorithm. The deployed AI (running in inference) is no longer trying to minimize a loss function or maximize discounted rewards.
2. This "desire" is not a "deliberate desire". It is more like an "animal desire" such as wanting to eat because of hunger. Aristotle draws this distinction. Such a desire is automatic (rather than deliberate) and is not part of choice-making.
Many of our higher desires are set for us (taught) by other humans: our parents and society. We wouldn't want an AGI that was able to set all of its own desires, any more than we'd want that in our children. Rather dangerous indeed.