In mathematical terms, you're stuck in a local maxima, and to get to a higher global maxima you need to descend the gradient a little bit and put up with some short-term pain.
Failing infrastructure can have disastrous impacts. It's not somewhere where you can afford to move fast and break stuff. There is no alternative to competent government here, no easy way out.
Things can get built fairly rapidly in many places if all restrictions are removed. You can definitely move fast and break things.
Unfortunately, these days exactions are also abusively imposed to offset the supposed costs of "gentrification" and other unquantifiable social phenomena, and sometimes the dollar value of such imaginary costs conspicuously set so high as to make a development financially unviable. This is how cities like SF force developers to include below-market-rate units or to make cash contributions to low-income housing projects, even when a project is not actually displacing any pre-existing tenants.