What's bad is if growth is achieved by irrevocable destruction of the environment, or deterioration of the human condition. It's not growth itself that is the problem, for there are many kinds of growth.
Growth in physical inputs? Energy use? Labor? Outputs? Value (GDP)? Some of these counter-balance the other.
If efficiency goes up (productivity), there is less need for labor, but it could also mean less physical inputs required, less energy, etc, so good for the environment. Also, the increased surplus means that labor can enjoy more fruits of its produce (in theory).
And growth in value is another thing entirely, because it is a human value assigned by our minds, not an objective physical quantity. Just take virtual/digital goods, someone can make one awesome piece of software, or a game, that the whole of humanity values tremendously high, but which has O(1) marginal costs to produce for each person, and which took relatively few inputs. If I suddenly uploaded the entirety of humanity into the Matrix, all of a sudden, value can continue to increase, without continued use of environmental resources beyond the energy needed to run it. Today, if I make a piano, I have to cut down trees and use tons of energy, so for everyone on the planet to own a piano would be very bad. But if I make a virtual piano, everyone on the planet can have one at relatively low cost, and then the question simply falls to how good the simulation can be.
I'd analogize Capitalism to evolution in an ecosystem. The entities within the system can grow, change form, multiply, and die. If they grow too much too fast, they can deplete their resources too quickly to sustain their population. They must have wild swings between mass die-offs and mass growth, or they might have a relatively stable pattern with low volatility, but while growth is slowed, they still have competition, they have not "switched off" evolution.
I don't think growth is bad if it comes from increasing efficiency, or an increase in value because of completely new inventions. Perhaps same physical inputs that could produce a device in 1950, produces something in 2010 that I value a lot more. It is the organization of matter, it's pattern, the information content of it, that can grow with infinite variation and which we can continue to demand.