It is true the prices can be manipulated over the short term - in 1979, the Hunt brothers made silver go from $6 an ounce to $48 an ounce before it crashed back down. Over the long term though, the value of commodities are tied to their value - how hard it is to find and mine gold, and whether people find it useful enough to exchange other valuable things for it.
If you were starving on a desert island I'm willing to bet you'd pay significantly more than $5 for 5lbs of potatoes if need be, because of their subjective value.
Likewise, people spend $$$ to mine gold because it's (subjectively) worth $90,000 for 5 lbs. If gold was as effectively rare as it is today, but easier to 'harvest' when you did find it, it would likely be worth about as much as long as all the other subjective factors still held.
Even today Saudi Arabia can pump oil for obscenely low prices (single digit USD per barrel), but it still sells near the market rate.
Also, I would recommend "Making Money" by Terry Prachet. Good book that actually explores a lot of the same ideas behind gold as the basis of a currency. In essence, the main character eventually realizes that the gold doesn't actually mean much, the true value of currency is that the society says x money gives you y goods.