It would have to be kind of fixed point of bugs (like finding cryptographic hash that equal to the input value). It wold be incredibly unlikely.

Yep. Imagine a prover that accepts all statements. It would accept a buggy program, and it would accept itself.

It would obviously help reduce errors but it wouldn't categorically eliminate them.

Only if the bug in the prover allows the bug in the prover, which seems unlikely (unless the bug is pretty huge).