Well, they can at the “real world” interface. They can prevent it from being used, but they cannot confiscate it.
Whether this is a problem or advantage depends on your point of view; The people of Cyprus, who woke up one morning to discover that any deposits they have above 200KEur were unilaterally borrowed by the government until further notice, or the Americans whose gold was confiscated in the 1930s just before the USD devaluation, might have a different opinion than yours.
Personally, I think it’s a good idea for government to have such a power if and only if a proper system of checks and balances is in place ; and the ones currently available are less than stellar.
For the US, for example, I would rather have confiscation power require 50 state AG mulitisig (or at least, say, 40 of them) - the fiat system, as is, effectively allows one person to drive confiscation.
Also, yours is an argument against any bearer instrument. Consider gold or well hidden cash - if you don’t know where I put it or what the serial numbers are, you can’t confiscate it.