No, that's exactly my point: the discussion that follows looks valuable, but isn't, because it isn't rigorous.
Job interviews are hostile. In fact, they're one of the most hostile experiences most of us have to deal with. I watch candidates in face-to-face interviews very carefully now, because this is a problem I am studying, and when you stop and consider candidates not as gamepieces but instead as human beings, you'll be amazed at what you notice. For instance, a majority of the candidates I've sat down with in interviews have, subtly but noticeably, been physically shaking.
Job interviews are like a game of Jeopardy! where one of the players is being filmed and broadcast on live television and the other player is playing from their couch and has been given an answer book, and the poor televised shmoe is judged on the same curve as the person on the couch.
Second, you're assuming that the way a candidate interacts with your team in an interview is an accurate representation of how they'll gel with the team down the road. Some of the shyest people I've interviewed have gone on to be the ones I actually end up talking to all day. Part of that is because interviews are hostile and make people shy, and part of it is that different people take different amounts of time to open up to strangers.
Finally, you're describing the "gut check" interview process used by almost every engineering team since the 1990s. We have evidence for how well that process works. Answer: not well.
As an aside: the idea of hiring candidates based on how well team members "get along" with them is repellant to me, as is "culture fit", since they're basically ways of making sure you have a team full of people who listen to roughly the same music, drink roughly the same beer, play roughly the same X360 games, and like roughly the same technology stacks. But that's an emotional argument; the real argument is that it doesn't work.
I think the problem is that everyone on HN thinks things are binary, when practically every scenario in this universe is not. You can't just eschew potential team fit because you want a rigorous hiring process that depends only on candidate qualification. The opposite holds true as well. Here are the two things you need for an awesome team:
1. A team that works well with each other.
2. Individuals on the team can do their job.
>>> the real argument is that it doesn't work.
It's not an argument, it's a statement. It would become argument if it would be supported with something that is not of equal weight with the contrary claim "no, it actually works quite well". Everybody has their anecdotes, do you have more than that?
Is your argument here that conventional engineering job interviews...
(the sort where candidates interview with a series of engineers from the team all of whom ask different sets of questions and seek to gauge from their answers previous accomplishments, working style, technical competence, and team compatibility)
... are generally an effective way for teams to screen candidates?
Or, ask an engineer how many times over the course of their career they've worked with "worthless" teammates, or people who should never have been hired. I've worked across a decent slice of the industry these past 19 years, and from what I can tell, this story is universal.
Another strong piece of evidence on my side is software quality, or the pervasive lack thereof.
Another piece of evidence is fizzbuzz.
Another piece of evidence is the (harmful) trend of demanding that candidates work on a contract-to-permanent basis as part of the recruiting process; what is this other than an attempt at work-sample testing that also happens to exploit candidates and repel the best of them?
I agree with you that close to 100% of working engineers are hired this way. But that doesn't prove anything; it's just as likely, perhaps more likely given the evidence, that what it shows is that development teams can ship marketable software despite terrible inefficiencies.
If you think that conventional hiring does a good job of selecting effective and competent programmers to the median ISV (and note here I'm stipulating ISVs, the best-case scenario for your argument, not line-of-business software teams at F500s), fine. I don't feel like I know a lot of smart engineering managers who are comfortable with the predictive power of their interview systems.
If I as someone interviewing for a QA role ask them a question such as 'Tell me about the QA work you are most proud of.' And they spend the next 20 minutes not talking about anything QA related it's a red flag. And an important data point to me when doing a retro with my peers. Perhaps that recent scenario is a reflection of the folks my company is able to attract in this area.
Conversely, if I am interviewing with you and I ask you, what's the most interesting thing your team has done and how did you help them accomplish it.' And you go into great detail about a deep area in your field that's part of the product your team works on its useful and important information when I need to make a decision. However if you are only interested and talking about me, my technical (or lack of - I know very little about security) understanding and you brush off conversational attempts to gain information that will help me judge the book by its cover it's usefully info when I need to make a decision.
I am not sure why your tone was so hostile? I did not mean to imply anything by my question. I am merely looking for other ways that people have found works for them.