Is that not the entire purpose of having the security guard though? If they are uncomfortable doing their job maybe they need to find a new one.
Guards know this too, and given the choice between letting someone through who you are pretty sure is legitimate and potentially starting a fight which will generate a complaint, they'll "do the best they can" and just wave people through.
Motivations interact in funny ways, and the parameters aren't like software: they're hidden and surprising.
Look at government and large corporate entities for how to address this. Visitor conference rooms are on the ground floor or away from the office environment. There’s a receptionist who controls access to the employee area, and identification is captured and associated with a staffer for both signin and signout.