There's an argument to holding on X money will let you make more money, therefore making your actual contributions larger / more valuable

You can more easily invest your second million than your first (Because you probably need that (Or at least a portion of that to live)

There's also no significant reason to start earlier (or later), unless you factor in dying as your stopping point, worthwhile causes aren't going anywhere

Because by then you've mostly finished spending money on raising your own children.