Your broker could change their T&Cs to enact a 1% conversion to USD fee. If you own 100 gold bricks, that's the same as them walking in and confiscating 1 of them. Presumably that's something that wouldn't happen if you owned the actual physical commodity.