It works great! I agree that we're unusual in the weight that we put in our references, but I've never had it be a blocker with an applicant. We run every reference through the same script of questions, for a call that tends to last about 30 minutes. Since many engineers often talk in interviews about what their team did, versus their own contributions, the third-party viewpoint is very helpful for getting that perspective. We've found references to be helpful in differentiating between average, good, great, and exceptional individual performance, and extremely helpful in understanding a candidate's teamwork. I personally find it vastly more useful than whiteboard coding and work samples for predicting how a candidate will actually perform.

Aside from their usefulness in making hiring decisions, as a manager, it has been great for jump-starting my relationship with my new hires with context on how they have worked in the past. That has been really helpful.

I agree with you that references have some limitations. We do, of course, expect that applicants will cherry-pick their references to make themselves shine. Yet, I typically get very candid feedback from the reference providers. I think most people simply aren't built to straight-up lie for someone else, even if they are a friend. References are typically someone in some level of authority someplace else, and they put their word on the line. Most people seem to take that seriously.

Another limitation is that people often can't use their current boss as a reference, and for people on an upward career trajectory, that may exclude someone who is capable of talking about the applicant's greatest career achievements. This was the case for me when I applied to my current role.

Lastly, we have to keep in mind that any single reference is colored by the biases and personality of its provider. And some applicants simply have access to better referrers than others for reasons out of their control. All we can do is make judgment calls when it comes to these things.

References from internship colleagues/managers, school mates or teachers can work a bit. But be aware that you will be getting very few data points.

Now, it is easier to influence new professionals than changing habits of seasoned professionals.

See my other response for more context, but we have a behavioral interview-style reference call script, which is universal for all roles at the company. But it still does tend to reveal good knowledge about the candidate's technical ability, even with generic questions. Occasionally, I'll add an additional question for tech context, but I find the basic script to be pretty comprehensive.

I'll echo the sibling post for inexperienced candidates. References from jobs in other careers also work. But to be honest, if someone can't find anyone to speak in-depth about their achievements and teamwork, they are going to have a really tough time passing the bar to get hired. We don't generally do work samples of any kind, but I'd probably have to make an exception in that case, to get some kind of picture for how they work.

We've certainly had a couple people not work out on our engineering team. But I don't think we've ever hired a total dud or a toxic person.

It's interesting that you mention internal referrals, because I think that can be a major source of bias. People feel obligated to put in a good word for their friend, and if I hold the referrer in high personal regard, my natural skepticism of them as a reference diminishes. I feel referrers should say their (small) piece at the beginning, but from that point onward, the hiring process needs to be independent of their influence.