I think this understanding is common, but it's totally incorrect and is part of the larger project to get fewer people to vote.
Voting does represent, in some sense, a decision of import for how society functions. A vote for a tyrant, for example, would be bad for everyone. But is voting more important or influential than the many, many other ways in which people support or oppose bad things in our society?
In the same domain - political donations support or oppose the same politicians (and the relative impact of a donation and a vote is debatable - at the very least people can donate many times!), but the discourse around them has none of the concern around "poorly informed" donators. In a different domain - working for a company can have an enormous good or bad impact on the world, but we don't caution people against "ignorantly" accepting a position (there is lots of discussion around the morality of companies - but people are not called "irresponsible" or "ignorant" simply for working for bad companies).
I would ask you to consider how big an impact a vote has and compare that any other decision we make that impacts the world. The narrative you're advancing holds voting to a much higher standard than other ways in which people affect the world. I personally think this is because the originators of this argument (but not necessarily you) want people to question if they should vote. However, they would never suggest that a person their suitability to select who to buy services from, or work for, or donate money to - because those are all actions taken by higher-income people and voting is (or attempts to be) universal.