Light rail is stupid. It’s a bus that can’t change lanes. A train that gets stuck in traffic.
And, as you said, they visibly disrupt drivers which generates class animosity.
It’s still at grade. Priority is meaningless if there is a car in the way when the guards come down. And those guards, in interrupting traffic, are annoying to drivers. (I’d also point out that the line between trams, street cars and light rail is ambiguous. It’s an American term describing principally European infrastructure.)
This possibility is so far outside my experience I can only think your perspective has more to do with emotion than logic. Maybe it happens more often in your city than mine.
It's still more reliable than the busses. I think it's pretty fun.
Either way personally priority bus lanes feel significantly more flexible and cheaper to implement than LR/trams...but that's just a personal opinion.