But then the rich would just choose to use their money in a way that only benefits them, while in reality, it might be better for the entire economy if for example more of the money goes to the unemployed.

Another issue that I see is that many people are very uninformed (or wrong) and if we let them choose directly where the money goes, it might be wasted. In the current system the way it should work (in my opinion) is that people vote based on the issues and values important to them and their elected then move the money based on researching and better understanding how to do it best (since that's the elected officials job, and they have time for that, unlike some of the population).