It's recommended to test dice used for fairness. Also - they would have to be really, really rigged to weaken the security in any meaningful way.

The same way that you can trust that we aren't brains in vats. At some point you have to have some trust.

You can use casino grade dice which are translucent specifically for this reason. And on top of that you can also test them yourself to verify.