Rhinovirus, common cold coronaviruses, metapneumovirus and even RSV don't commonly cause the high fever seen in typical symptomatic influenza cases--which is what the influenza vaccine is seeking to avoid.
And the whole point is that vaccination will help to prevent a symptomatic infection typically characterized by ~103F fever for up to 3 days with one of those asymptomatic infections that you mention.
And the problem is that most adults may go years or even decades between those severe enough flu infections. They're mostly likely just catching colds. Maybe they caught a flu strain and their adaptive or innate immune system fought it off so they really did have "24 hour flu". But the point is that those very mild respiratory infections are not characteristic of a typical symptomatic influenza presentation which can be prevented by the vaccine. People become complacent talking about those symptoms as being typical of influenza, while typical symptomatic presentation of influenza is much worse.
And even if you want to go down the rabbit hold of arguing PCR this or that and maybe there's a wild amount of asymptomatic influenza infections every year -- it doesn't really matter -- the point is the severity of the infection on the higher end of the distribution. And rhinoviruses really do lack that level of severity because they don't carry the same disease burden and don't kill as many people.