Gold has been the longest most stable 'currency' historically. Governments with some of the largest economies hold large supplies of gold (and they're not trading it for BTC). It's value has been on par with inflation since we've started keeping track of its value. This creates confidence among buyers.
People agree it has value because the most powerful players in the global economy agree that it has value. There is also a limited amount of it.