My reasoning is that a clock is either right or wrong.

The average of a right and a wrong clock is wrong. Half as wrong as the wrong one, but still wrong.

If this is a good mental model for dealing with clock malfunctions depends on the failure modes of the clocks.