The manager's behaviour is absolutely appropriate, and not impolite.

Also, someone with antisocial anxieties like this probably would have difficulty with that kind of confrontation.