Money is not speech, and courts do not treat it as such. The most famous recent case,
Citizens United didn't even flirt with the notion. (Despite much commentary to the contrary, apparently from people who didn't read the decision.)
Money does, however, enable speech, and restrictions on things which enable us to exercise fundamental rights is at least potentially a restriction on that right. That's the framework the Supreme Court used.
If you disagree, consider: Pick a right you quite like (gun ownership, abortion, speech, voting in an election, etc.). Clearly a law banning that activity outright, or severely restricting it ("you can't obtain an abortion", "you can only speak in favour of Trump", "you can't own a handgun", "you can only vote for the Democratic candidate") would be absurdly illegal. It follows that a law preventing you from spending money on that right ("you cannot pay for an abortion", "you cannot spend money on criticising Trump", "you can't purchase a handgun", "you can't spend money travelling to the polls unless voting for a Democrat") would also be illegal (if severe enough at least).
The scope in which government can make your ability to exercise fundamental rights more difficult is non-zero, but sharply proscribed (witness the long string of court decisions striking down red state abortion laws).
It's possible that the restriction at stake in Citizens United was so minor that it should have been permissible, but I take your argument to be that it was anything but minor. In which case, the restriction probably was unconstitutional. In the US system, speech is protected, and (like it or not!) political speech is the most protected form. Any law which meaningfully impacts any real person's ability to speak about politics is presumptively unconstitutional, by design. And the more meaningful the restriction, the more that applies.