Everyone's threshold for "serious" harm is different.
For adult men, if you drink fewer than 15 drinks per week but greater than zero, the expected decrease of average lifespan is measured in months not years. Obviously more than zero. Is that "serious" enough to not partake? Up to each person.