Between the Mason-Dixon line and the opposite latitude in Brazil it was more Malaria, Yellow Fever, and other mosquito born diseases rather than smallpox that did most of the killing. Malaria alone is responsible for about 5% of all human deaths through history, compared to something like 1% for smallpox.