This is most common in fields where a Ph.D is required to do meaningful work (e.g. biology, philosophy, etc.). It is especially common in the humanities, where many (most?) suffer from imposter syndrome (in many cases, rightfully) so this gets tied up in ego issues too.
On the receiving end of recommendation letters, I can say two things:
1) They're almost meaningless. Most are glowing. 2) They reflect more on the writer than on the student.
That's something students don't (and based on information they have, can't) realize. If they pick the wrong professor to impress -- someone like Agnes here who applies a philosophy of "hyper-rationalism" to recommendation letters -- their career is shot.
For the most part:
* If you're applying somewhere, I recommend talking to many recommenders. Ask whether they can and will write a strong recommendation. If so, engage with them. If they don't answer, don't take a recommendation
* If you're looking for a professor for graduate school, talk to their students first. Look over where they are now. A state school with a good advisor will beat an ivy league with a bad one, hands down.
* If you're looking for an undergraduate projects, do the same. The stakes aren't as high, but you'll do a lot better if you chance on someone who cares about you as an individual and works to make you successful (which is their job, as a professor) rather than someone who places their career first (which is the reality, of most professors).
If you want to take a hyper-rational approach to it, think of it this way: if better students students pick ethical, humane professors, you're giving ethical, humane professors an edge in the academic rat race. And you're giving sadistic bastards a disadvantage in the rat race. You're making the academy a better place.