Plenty of men aren't there and don't feel guilty about it. And stigma? Only if you live in a very small, close-knit community. For men, unlike women, involvement is a choice. Not an easy choice, but still a choice.
The biological consequences, on the other hand, are certain and universal.