Well I'll say it then, you really should unfollow more people. To a point, the more you unfollow the better the signal to noise ratio improves. You just have to find that point.
It isn't crude or destructive at all because you will see those good quality tweets from people who are heavy users. You will see them when others who you still follow retweet the good ones. Your feed should be a solid mix of tweets you curated by following interesting people, and tweets those people curated for you by retweeting.
It can be a bit awkward to unfollow people you know because there is that social pressure to 'follow back' but this idea is worth it for the positive results.
To give a specific example, try unfollowing all local media outlets (newspaper, TV, radio) and instead follow one or two reporters whose feeds you find interesting. You will still see all the interesting local news and you avoid all the marketing and fluff that pollutes the media outlets' Twitter feeds.
I read every single Tweet in my feed every day and it only takes a few minutes. There is usually a ton of interesting and relevant stuff in there. I couldn't do this before, when I was following too many people.