A downside of that is that it can intensify populism, since you are essentially always campaigning, never governing. One person wins, but instead of getting a few years to try their agenda, the opposite party is every day looking for openings to get them booted out of office early. An in-between idea is just the classic impeachment. It can remove someone from office early, but is supposed to be used only for serious breaches of trust or lawlessness, not mere political disagreement. If Obama totally went nuts, Congress could remove him from office, if they weren't also on board.
Overall I think I'm more worried about constant campaigning (and the amplification of political advertising and PACs that entails), so these various measures strike me as more worrying than the problems they try to solve.