This is the good ol' Trusted System issue. I've roamed in this area before, both in terms of creating a chain of trust in crypto systems and also at a more philosophical context for voting systems. I'll decompose the issue into more abstract questions:
> Can we ever trust any system?
Yes, to an extent. There is no such thing as a system that can be trusted completely, but we don't need it to be in 99% of cases. One might say "you can trust crypto primitive XYZ. If you use it, it would take 1 billion years to break". That might be true, but side-channel attacks, leaks, statistical biases and whatnot will always be an issue.
To get as close as possible to trust in a system, it needs to be formally verified with proofs. That's the best we can do program/algorithm wise, but even if we trust the program, it cannot trust the system it resides on.
> How can we achieve trust then?
You know how bitcoin is based on a distributed consensus algorithm? It protects the whole system from collapsing due to a bad actor in the system. Even if thousands of people decides to cheat, it won't have any considerable effect.
Let's say you buy a hardware wallet from a reputable vendor - if they decide to cheat, you will be at their mercy. To combat this, you need a way to verify that what it does, it does so correctly, but also without side effects.
This is again something that needs to be formally verified. Any deviation from the spec will stand out like a sore thumb. To achieve this, we need to introduce a verifier.
The verifiers job is to check if the hardware wallet did it's job, but without being in the possession of the private key. There are lots of ways to do this, but a hot topic today is zero knowledge proofs, where the wallet would need to stand up to scrutiny.
The verifier would also need to check the results on the blockchain. Not just that the result generated is correct, but also that it is without side-effects.
> But then we have to trust the verifier!
Yep, and each time we introduce a verifier for the verifier, we will have made the system more trusted. Let's say we have N verifiers, whos best interest is that your wallet did the right thing.
In a transaction, it is not only in _your_ best interest that the transaction is correct (and without side-effects), but also the other party. We can extend this system to be a small group of people in _any_ transaction. If a small group of verifiers all agree with a certain level of consensus, then we can trust the system beyond a reasonable doubt.
This might sounds familiar to those who work with blockchains - and you would be right. It is eerily similar to how it works today. However, the blockchain covers only the cryptographic guarantees. The system needs to be extended to cover formal verification of the system as well.
> Example
Formal verification is a mostly academic exercise for most, so I'll give a small example for those of you who are unfamiliar with it.
Let's say person A and B make a transaction. Both have super secure hardware wallets and the crypto used it state-of-the-art. It should be secure right?
We can review the code of the system, but it is hard to identify mistakes. Who knows, maybe there will be a new area of vulnerabilities in a few years, and we never saw it coming.
Within the area of "correctness", we first need to make a formal specification. We create some testable properties about the system that needs to hold true, no matter the transaction or who is involved (these are called invariants).
So person A transfers 1 bitcoin to person B, they do so by signing a nonce with a private key. Person A checks the nonce and ensures it is indeed random (test 1). The signature is sent to person B, which then tests if the signature is no different than random data (test 2).
Howe test 1 and 2 are performed are incredibly important and very difficult to do, but not impossible.
If test 1 or 2 happens to be non-random, then we can just reject the transaction. We don't know if it was non-random by chance or on purpose, but since it does not live up to our criteria, we will reject it.
This means Person A will check what they got from person B and vice versa. However, why not have a bunch of random people participating in the block chain do the same checks?
If 0.1% of all in people in the blockchain checks the transaction between person A and B, and they all have a say if the transaction gets rejected, then we can trust the system beyond a reasonable doubt.
And no, this system is not perfect. We don't need it to be. We need it to be good enough so it becomes incredible hard to cheat. Also note that I've omitted a lot of details for brevity as well.