Counterfeits, adverse grading, appraisers swapping your stamps/coins out for lower grade ones, and simply lying to you about the stamps/coins.

It's like the art world. Full of fakes, fraud, etc.