Something about this argument doesn't smell right.

I care about the crime rate, or car accident rate, or botched surgery rate, because it tells me the odds that something bad will happen to me.

Given a stable rate of bad things, population growth alone causes more bad things to happen.

If we took your argument about absolute harm seriously, then we should, say, encourage population reduction until there are only 10k human beings on the planet. That would surely minimize the number of bad things that continue happening to people.