I'd want to whitelist specific people before they could send me a calendar invite. Every other invite request should never reach my device. If I don't even know you, why would I want your invites anyway?
Boss: Why aren't you in the meeting with our vendor to upgrade our X system?
You: Oh I whitelist all my invites. You see, I am thinking about security and don't want to receive invites from someone I don't know.
Boss: Clear your desk, security will walk you out.
ExternalUser: Hello here is a calendar invite I would like you to attend, please confirm or deny
User: Thank you, now I can verify the request and choose to add this to my calendar or not
If I work with them, I would have them whitelisted. If I've never even heard of them they have no business sending my devices calendar invites.
Boss: Why aren't you working on that project I gave you?
You: Some stranger in Indonesia invited me to a sales meeting instead.
Boss: If I need you to go to a sales meeting with someone from Indonesia I'll tell you to! Clear your desk!
If someone is going to make some demand for my time, the very least they can do is give me notice outside of my icloud calendar. An email, an IM, a phone call, etc are all very easy and they allow me to make sure it's real before it has any chance to interfere with my schedule. "Hey Boss, this guy says he's our new IT guy and he wants to talk about my network settings" or "Hey $vendor, I just got a call from $rando saying he's our new contact, can you verify that for me before I tell him everything I know about your propriety applications?"
It helps that I like to keep my work devices and my personal devices entirely separate. If someone in the office wants to pull me into a work meeting through outlook, they'll already have to have an account set up on the company's exchange server. Anyone outside of the company I should already have a relationship with or at least a heads up.