> it's nice to be relatively young and healthy when you're chasing toddlers
Why the assumption that you'll be not so healthy in your 40s and 50s? My parents had me in their late 30s and they didn't seem to lack for energy compared to what they and their friends remember of their 20s. Hell, I'll bet even if they had more kids right now they'll still have the same health and energy.
Eh, on this point, I'll relate a comment I heard through someone recently. They were talking to a family member, getting their thoughts on the trade-offs between having kids younger versus older. Their answer was wonderfully succinct: "You're trading energy for patience."
The rest of your points are definitely valid. I have plenty of friends who've had kids later in life, naturally, without spending money on IVF, but there's definitely increased risk.
As for Downs, the magic number seems to be 35-36 years old. Even then, the odds don't go to 100% but they definitely increase substantially, so it's something to keep in mind (as well as your feeling on the ethics of prenatal testing and the associated decisions if they come back with a positive indication...).