I'd argue that there were three.
The first, and unquestionably illegal, was what Tom Hayes did i.e. collusion between actors at multiple banks to move Libor. I don't think he should have got the sentence he did but prison was definitely justified.
The second was swaps traders getting their own setters to go in high or low. I'm not at all sure this was illegal. Given that Libor was completely broken at the time, I'm not even sure I'd argue that it was unethical. As there was, essentially, no uncollateralised inter-bank borrowing, any figure was wrong. And, as you say, the effect was minor and not particularly directional.
I think your second point, the systemic low balling of Libor by senior management, is, by far, the most important. It's also the one that has had the least public attention. What is particularly problematic about it is the influence that governments had. Because of the latter, I'm sure that it won't ever be properly investigated.