This is a specious argument. Using your logic I could say that this is because everyone agrees to use it to pay taxes. Then further conclude that the government is only legitimate because people think it is and they have not deposed of it. Therefore both the dollar and the United States only exist with the consent of the people. ie, because people agree that it provides value.
However, what money actually represents is debt. This is the same with any asset class, but especially with a country's currency. If I have a goat, and you have a cow, and I want milk but you're not interested in what you have to trade I could issue an IOU, if for instance I have excess feed for my goats that I could trade in the summer. This sort of arrangement could be made in smaller communities, because breaking the social contract could have dire consequences. However, when you scale that up on a societal scale you begin to create a currency. The name of the currency doesn't really matter, honestly, it could be the dollar or bottle caps, as long as it's a currency that represents value to everyone. In a small community a hundred bottle caps could be worth the value of a pound of wheat, or 100 pounds of wheat.
How do you decide? Well it all comes down to the person who is buying the product, but then again, not really... It's also a social contract that people are making. The value is defined by making comparisons both by how useful the asset is, how rare the asset is, the demand for the asset, but how much people agree the asset is worth collectively and in order to make such a decision the people have to decide how much value a currency has when exchanging goods and services.
The same with any fungible asset - all value is created because people collectively decided to assign it this value. You can see this actively playing out in any asset class from housing, stocks, and cars, to things as trivial as magic the gathering cards, to things as important as the value of one's labor. When you're done at the end of a job someone will say to you "how much money do I owe you?" not "how much taxes do I pay you?" because therein lies the real value of a currency. The labor, and the asset that is being provided is the value, not the taxes. Without a common collectively agree upon asset that people can exchange then there is no way to exchange services without a bartering system.
The dollar's power is that universally everyone agrees that it has value. However, the dollar can still have power outside jurisdictions that don't pay taxes to the United States government. How does that square with your argument? Why does it have value? People in France are not paying taxes to the Government but they can freely trade Francs in exchange for Dollars. So it can't be taxes. If there is no currency that everyone agrees has a common value how do you pay taxes? How do you even decide what a fair tax value is? If I say tomorrow that the new currency is bottle caps, what is the fair tax rate? Well, the first question you would ask yourself is what is the value of a bottle cap? And thus you return to the beginning of this post, and society has to begin again to make another unspoken agreement about the value of a new currency. The value of a currency is again will be decided relative to the perceived value of the goods and services provided, and the agreed upon value of said currency.
In my imagination milk is worth 100 bottle caps, but you may disagree. Thus we barter, and therein lies the value of all currency. Bartering isn't gone, it's just invisible, or symbolic nowadays. So when you're in a business meeting trying to sell your next startup imagine how much milk you can buy, or whatever.