So the most ethically optimal population is one in which adding new members will cause life to no longer be worth living to at least the same number of other members. The effects of each new person tends to push another person into a suicidal state.

I think the problem here is in using happiness as the thing to be optimized in ethics. A better measure may be the odds of survival. A maximally large but miserable population may have lower adaptability, and thus survival potential, than a somewhat smaller but happier one.

The most ethical population becomes the one that can increase its own evolutionary fitness the most efficiently, and that is not one so resource constrained that growth creates equivalent death.

Optimizing for survival, we optimize for the population's potential to explore its fitness space, a kind of computation. If you design a computer cluster to make this calculation, you would choose a smaller number of highly capable computers rather than a larger number of barely functional ones.

Optimizing for survival is something like optimizing the cumulative parallel computational capacity of a population rather than its cumulative happiness. The fact that it may imply greater average happiness might just be a happy side effect.

It would also tend to lead to a larger population in the long run, and thus a greater cumulative happiness in the fullness of time, considering future generations ... which would make the two measures eventually equivalent.