The only downside I can think of is that links to your website that are shared on Facebook / WhatsApp / Messenger won't be able to show a rich preview because they won't be able to access the OpenGraph tags on the shared page.

Rich previews are known to cause higher clickthroughs than non-rich previews (if you care about that).