What would it take for you to have two more children starting today?
Not just one more child, but two. Not the Japanese, but you. Not sometime, but today.
I think that just limiting the horizon on this issue of childrearing is not as helpful. Thinking of the concerns of the 'other place' makes it less impactful. Thinking of just one child more is less impactful. Thinking of a future date where everything is perfect is less impactful.
Making the problem bigger, immediate, and personal may give better insights.
My personal answer is health, time, and then money. My family's health for two more children is not the best. Another pregnancy, starting today, would endanger us. Then it comes down to time. With the family we already have, we're already time strapped. A live-in-nurse would be hugely beneficial to us to have more kids. Basically, we'd need a 'wife' to get all the housework done. Then it would be money. College in the US in ~18 years is going to be very expensive. Kids are expensive. Daycare is another mortgage as is.