But this is the problem. It means that a party being illegal isn't such a big deal in and of itself. It's only a problem if it challenges the mainstream parties. You can see why people would think it's just a political maneuver if they're trying to ban X party only when many people vote for them. Bonus points for this party portending to "represent the voiceless" or whatever.
If some party is illegal for whatever reason, it should be banned right away. Just because you can ignore it doesn't mean it should be ignored. It would also be much easier to prevent having a sizeable chunk of the population vote for an "illegal" party and the headaches coming from that once it becomes big enough.