The difference is that nothing can remove the confidence behind gold - not even forced confiscation by the state (which just results in hoarding and buried treasure).

However, it's easy, simple to undermine confidence in fiat currency - just overissue and prevent conversion to other forms.

Can confidence in Bitcoin be removed? Not sure about that one. Would banning trade in it like gold can be banned remove all confidence? Very hard to say, and something to watch.