I am sorry to hear this. As a recent father myself, I find the fact that comparatively few fathers here in the US do (or can) take this leave very sad. The act of caring for our newborn is how we form our attachments and love, and it is ultimately a loss for the dads that choose not to participate in the caring. My dad never really took care of me as a baby (in the feeding, diapers, etc. sense) and is squeamish/ not able to do it with our own child now. Again, a loss for previous generations of men.
Which is all to say - if this company in the slightest gives you more negative feedback about taking leave,.. I would, if I were in your shoes, be looking for a new place. They're not showing you any loyalty, so you don't owe it to them.
A twitter thread from Ezra Klein (also recently took parental leave) that might resonate with you: https://twitter.com/ezraklein/status/1457771021327503360