Your absolute rights end when they encounter another person.

As society, we've decided peoples individual rights can/should be curtailed when they harm others. We will never agree on definition of "harm". But, certainly money can be used for harm. From I pay someone to kill you, to I donate money to group government has labeled "terrorist", to we made profit polluting environment, etc.