I'm not completely sure what the answers are here yet, but I do agree that it can be very psychologically reassuring to know that the only people involved in a conversation are the genuine invitees.
That said, abusive people do exist and are a legitimate problem, given the damage they can cause. Their abuse may be overt (threats, violence, noise, etc), or it can be subtle (for example, manipulation over long periods of time).
Some of that abuse may come from prior anger and frustration outside their control, and perhaps it's good to allow people to let that out -- as long as it doesn't end up harming other people in the process.
Would the situation be improved if the service provider could only step into the meeting when explicitly requested by participant(s)?
To follow your analogy, that could be seen as the equivalent of someone experiencing a medical emergency during dinner at your house and requiring outside assistance.
All these options would be gamed and misused, as they are during existing use cases in real life. Some people over-react, many people under-react, and society itself changes so it's important to build in flexibility for transparent and accountable change.