You're right that material things can weigh you down. However, in the case of stamps they're probably one of the smallest and lightest physical items one can collect.
However, they aren't stored in an optimal packing, because that would ruin the stamps. They're stored in binders, increasing their weight and volume more than tenfold.
Yeah 10 times a small number is still small.
If you have a one-stamp collection, you're absolutely right. But collectors often end up with many boxes of binders.
Element collecting, as practiced by Bill Gates, might be the lowest. Although collectors tend to horde 10^23 items more than needed.