As you said, communication is a necessary skill. Even in a world where email, text, etc are becoming normal means of communication, you need to be able to clearly communicate on a regular basis, extemporaneously and face-to-face with another person. Your ability to communicate your weaknesses and strengths well in an interview is absolutely essential to anyone during the hiring process. Quite frankly, most companies don't care if you're the absolute best in your technical field - if you can't match the culture of the office, you're only suited to remote freelancing - if that! Remember that HR has to constantly cover themselves - every employee is (potentially) a liability waiting to happen; if they hire someone who contributes to a negative/toxic work environment or is a sexual harassment risk (either as victim or aggressor), they're likely going to be fired for introducing that risk to the company. You have to tell HR in language they understand and can quantify that you are not a lawsuit risk.
The point is definitely not about prepared presentation, it's about being able to work and exist with other human beings like what HR calls "a normal person" - which is really a person with no negative qualities that will potentially endanger the company. Believe me, I've had a few red-flag interviews where the person came off very rude when I asked them about their personal and professional weaknesses; or when they had an issue with a manager. I've also had interviews that felt 'weird' because the person seemed to fixate on the gender of each of his previous managers.
Also, realize that some of these questions inform how interviewers view your references. If you list a certain manager as a reference, you're likely to be asked about a time you had a frustrating experience with that specific person as a manager - oh, and as a side note, if you're asked for three references in a tech field, do everything in your power to include one opposite-sex individual. When we call to confirm the reference, we will in turn ask the reference about that same incident with you as an employee - what happened, what caused the situation, and how it was resolved. We can't do a proper 360 evaluation on you, but we can certainly try to see a more full picture from the people you mention as references and their answers to questions about your conduct.
Your technical skills are either there at the level the employer's needed level, or they aren't - in the latter case you got filtered either by the ATS or someone like I used to be. It's the interpersonal skills which aren't well-described by a resume.
One other thing - "He's a great guy and really good at resolving interpersonal challenges" is a stronger internal selling point from a recruiter to an HR hiring manager than you'd immediately think. Remember, the default for the company is that you're too expensive - some recruiters (internal and third-party) have to justify not offshoring a role before they can even begin suggesting candidates. If you're not going to contribute to the office positively and the HR manager thinks you're going to be a challenge or a danger, you will not get hired.