You need to have a perfect track record. You have to have worked in a couple of different places, with no gaps, you need a lot of publications and "visibility". And even then, in my experience about 50% of positions are given on merit, and 50% are given seemingly randomly. Even if you are incredibly self-confident and count yourself among the top 5%, it is still a gamble.
Academia works like a guild, or maybe like the military (up or out). But it seems everybody would benefit if hiring was changed to be more like outside academia. No more temporary positions vs unfireable professors, but instead hire everybody on regular open-ended contracts. Don't insist on an unbroken career and years of travelling journeymanship, but value broad experiences and career changers. I know a lot of people who are considered academic failures but were welcomed in industry "with kisses".
I don't think the cause is just resistance from entrenched people, or ideology. There is a lot of neoliberal pressure to run universities more like companies. And I guess hiring is one of the few areas where this would not be a bad thing. Is it just shortsightedness then? Or lack of funding, and scarcity of jobs?