You're mixing up different "you"s. If the American legislature got together and passed a law saying the American people just don't want to do so much foreign aid anymore, that would be a hard call.

But that's not what happened. Elon Musk, a random rich guy who was not himself financing the charity, appointed himself dictator of all American spending programs. He promised his patron that he would make the government run more efficiently, but found himself unable to. Then he went around randomly breaking charitable programs in an attempt to prove that his failed government efficiency initiative was producing meaningful outcomes. That's why he is accountable (and will be held accountable) for the people his decisions have killed.