Mass in the universe appears to be (very) roughly uniformly distributed, so even if there are large bodies of antimatter far away in the universe there would have to be a transition boundary somewhere between here and there where the universe goes from being mostly matter to being mostly antimatter. The universe is big and stuff would sometimes cross this boundary and get annihilated, and if this happened it would be the brightest thing in the sky, briefly outshining entire galaxies. We’ve been watching the sky for a while now and have never observed a bright visual event with the spectral signature of a matter/antimatter annihilation, so we assume there is not such a transition boundary, and by extension that the universe is made up of mostly matter out to the edge of the observable universe.

There's a great episode about this on History of the Universe yt channel - https://www.youtube.com/watch?v=xJGaqe5t14g

It talks about symmetries, but has a nice story about this exact hypothetical scenario. (Someone else already replied why this probably isn't possible in our observable universe, but the episode is cool so I thought I'd share)