I agree, the disease itself might not have killed as many, but the effects of the pandemic most likely did. It was by far the largest deviation from the norm during the time and therefore carries by far the largest responsibility of excess deaths (as excess deaths are deaths outside of what would be ordinary).