Maybe this is the wrong mental model, but I picture it like a hard disk slowly filling up with bad sectors. Except in this case, the disk controller can't tell you how much free space the disk has.
So without the knowledge of how much cash I'm hoarding in my closet, how can you accurately value USD? I don't know how much of the answer is “you can't, exactly, but there's enough other trading that it doesn't matter” (which could apply to Bitcoin as well) versus “the value is fixed by external factors such as the US government”.
The are many problems with such systems, which is why the entire planet moved away from them starting about 100 years ago. Just like the initial users of BitCoin who are sitting on millions of the things are limiting the bitcoin supply in order to slowly sell them for real coin.