Life on Earth consists of Von Neumann machines. That's exactly what life is. Yet our life has not converted the entire mass of Earth into itself, and there's still plenty of places where you can't easily find signs of life. And our life has had billions of years. The assumption that Von Neumanns will very cleanly and exponentially increase in number until they are all over the galaxy is too ideal.