The way to prevent a small minority from deciding stuff like brexit is by having a minimum of attendance required - if not met, the results are void.
Apathy can be driven from a feeling of helplessness as well as simply not really caring or knowing what's happening.
Additionally, if people had to vote, would they try and engage more?
I guess the key question would be, would the ill-informed that don't vote cause enough of an issue to outweigh the benefits of having higher participation from those who feel simply more left out?
I would love to see other propositions as well, currently a candidate/party has two main actions that are beneficial for them:
1. Make more people vote for them 2. Make fewer people vote for the other side
So weirdly our politicians are rewarded for lowering participation in groups that don't vote their way.
What if we could link the control with the total turnout? Perhaps at least with some things, like to force a bill past the Lords requires enough MPs to represent more than X% of the country itself. Maybe this would be a terrible idea, but I'd love to know what other alternatives we could consider.
My explanation for why someone might rationally decide not to vote did not reference or depend upon the extent to which the person was informed or how much they would exert effort to be better informed. Some of the people who don't vote now are well-informed. They don't vote because their individual vote won't change the outcome. You want more of those folks to vote.
There are other people (who you're talking about) who aren't well-informed, but would be forced to vote in any case.
Your assertion "Mandatory voting just results in on-average less informed voters" would be true if the two groups I mentioned were, on average, less well informed than the people who normally vote.
That might be true, and it might not be true. Do you have strong evidence either way? Or just a hunch that if someone bothers to vote they are more likely to be well-informed?