I'd say, you can (Bitcoin is one of many recent exercises of replacing trust with math). But it's not worth the price. Or, put another way, trust is a ridiculously powerful optimization that enabled humanity to form societies in the first place. It's very similar to how introducing a central node into a network reduces communication costs from O(n^2) to O(n). In the general case, it's stupid to not take advantage of it.
I consider Bitcoin to be the closest we've came so far to expressing trust in units of energy. POW shows us how much computation has to happen to achieve mathematical guarantees in lieu of trust. Somebody could probably derive some upper and lower bounds on the energy costs of trust in terms of information theory. I'd love to read such a paper.