So if the population grows from 1 (in units of 8 billion, say) to (1 + A), and the productivity per person goes from 1 to (1 + B), then the world economy goes from 1 to (1 + A) * (1 + B). If the population doesn't grow, then the economy only grows to (1 + B).
As you say, so what? Is the world economy some kind of deity, that we are here to serve its needs rather than our own? If each of us are the same (1 + B) better off, why do we care whether the global economy grew more or less?
As Edward Abbey said, "Growth for the sake of growth is the ideology of the cancer cell." If the growth actually leaves people better off, that's a positive. But if it just makes bigger numbers on a page, what's the point?