First off, yes, I agree with the notion that you should take the leave, and that being with your children at this time is much more important than work. That said, here is how I've seen other folks handle it:
1. If possible, especially for paternity leave, I think it's a good idea for both your team and your family if you can break up some of your time, e.g. take a month or two now, then come back to work for a month or two, then go back off for a month or two. Many folks have family that will help with the kids in the early days, so it was helpful to always have a extra pair of hands at home for the mom during the first half year or so.
2. Your team will feel the burden of extra work when you're on leave, but I think it's good just to recognize that and set things up as best as possible beforehand. Sometimes people get sick, or people leave for other reasons. Mat/pat leave is just one more thing people and businesses need to learn to handle, and (and I say this as someone without kids), even a small team will have to learn to deal with these "curve balls" at one point or another. You shouldn't feel guilty for taking your leave, just like one shouldn't feel guilty for getting sick.