For a particularly dramatic example of this, read G.K. Chesterton's _The Napoleon of Notting Hill_.
In the book, England, late in a long and uneventful history, is governed by lot; a randomly-chosen individual is crowned King and given dictatorial powers. Eventually, the lot falls on a bored nihilist with a fondness for 16th-century uniforms, and within twenty years London is a patchwork of feuding principalities with antiquated weapons, beautiful heraldry, and a whole lot of innocent people dead.
This book confirms what you're saying: it's not very likely that a barking madman will be randomly chosen as dictator, but it only has to happen once.
(The strangest thing about it is that Chesterton saw the most troublesome belligerents, who at one point threaten to use a water tower as a sort of late-Victorian atomic bomb, as the good guys; their willingness to use violence brought beauty and dignity back to a pacific, indolent world. This book was written before Chesterton became Catholic, and its bored nihilism reflects his own feelings at the time; I strongly suspect that it had an influence on the development of fascism, given that Mussolini was a fan of Chesterton's.)