> Surely it's up to me to accept whatever I want?
It is, up until the point you turn it into a debt. This means you can require trading apples in return for potatoes, but you have to do the transaction up-front. So if you ran a food stand and rejected any cash transactions at the point of sale, you are completely within the law.
You could not give your apples to your neighbor in the spring, and force them to settle up via potatoes in the fall. If they wanted to, they could pay you the equivalent of their potato debt in cash.
And yes, if someone ends up owing you money and offers you cash to settle - you must accept it, with very few exceptions. All debts public and private actually does have some serious teeth after all.
A long time ago I had to settle a bet on this topic, and it was more extensive than this. The tldr; is that at no time in US history have you been forced to take US currency, with the exception of debts.
It actually does make sense. What happens in the event your neighbors potato harvest fails and they get out of the business? If you sued them in court, the court would award you the market value of the replacement cost of the potatoes you did not receive (most likely). And thus they would be settling up via cash either way.