One of the arguments on this topic is if there even is an observer effect at all, with one alternative being that the quantum wave function never collapses, instead it is our own wave function becoming entangled with that of the particles we observe: the part of us which observes |A> is correlated with the part of the particle wave function in state |A>, the part of us which observes |B> with the part of the particle wave function in state |B>.
(And I'm also oversimplifying, because I’m still learning this stuff).
I think it’s something like: yes, everything is “seen” by the rest of the universe, but “seeing” is done with discreet did-or-didnt happen events (mostly photons) which can be suppressed.
Still though, the fact that some states remain indeterminate until that interaction takes place means the universe holds its cards pretty close.