But that wasn't when we were at our weakest, not necessarily. Diseases need to spread. When we lived in scattered hunter gatherer groups it was really tough for diseases to spread, and if they couldn't spread, it was difficult for them to persist in a population and evolve. Now the spreading is easy. Furthermore, spreading is easier among certain animal populations because we raise them packed into warehouses and we ship them around the world via truck, train, boat, and plane.
The world is a very different place, so using the past as a model to predict the future is dicey.