The epistemological reality of those events are not as simple as you assume. For the most part, that annihilation event is a short-hand analogy for a term in an equation. Another way to think of it is that this 'happening everywhere' style of particle-anti-particle annihilation relies on the inherent uncertainty of energy over short time scales, and thus it 'borrows' energy that doesn't exist in order to precede, but under any measurable timescale the energy must be conserved, and so these events are likewise unobservable. So it's not a simple matter to assume they happen, or that they are the same phenomenon as annihilation in general.

Look at the PBS Space Time video previously linked in other comments. It will help.