Life on Earth consists of Von Neumann machines. That's exactly what life is. Yet our life has not converted the entire mass of Earth into itself, and there's still plenty of places where you can't easily find signs of life. And our life has had billions of years. The assumption that Von Neumanns will very cleanly and exponentially increase in number until they are all over the galaxy is too ideal.
And that most if not all probes in every generation will even find something other than empty space because the universe is very very very big and the likelihood of traveling in any direction and ever hitting anything is very very very small.
Although if you assume the universe is simple enough to describe on the back of a napkin then it probably is straightforward.
ADDED: There are also questions of resources, motivations, and e.g. whether we'd even want to build self-replicating probes.