Legitimacy is fundamentally about trust. Trust that the elections were fair, even if you don't like the outcome.
If your candidate won, your opinion on the legitimacy doesn't matter much. If your candidate lost, your opinion matters more. If you think that the elections were legitimate, your opinion doesn't matter much. If you think they were not legitimate, your opinion matters more.
It doesn't really matter if the elections were fair. If the losers don't trust the system, the elections were not legitimate.
A society can handle a small number of people who question its legitimacy. Maybe 5%, maybe 10%, maybe even 20%, depending on the overall level of trust. If too many people don't trust the system, the society doesn't really work anymore. Laws, constitutions, and other institutions are only as strong as people's faith in them.