It needn't be most, or even many on HN, and people of all kinds vastly overestimate their abilities. It's just that on HN it's overestimating with great ambition.
(I say this very confidently, don't I?)
Is it not Occam's razor that people are like this because this world of startups, "cutting edge tech", "move fast and break things", etc. gives quite clear incentives to be like this? The entire of financial world of tech is quite significantly propped up by the inertia of unearned confidence!
And then you become the richest man in the world and buy Twitter and show everyone that you're kind of just clueless outside of your area of expertise, but putting up with you is profitable enough that people just go with it.
People can do this repeatedly with positive feedback and increasing scope until eventually it doesn't work.
What I think happens is people who are very knowledgeable about a subject are hyper-sensitive to slightly incorrect information. And to boost their egos they like to diminish the people making the incorrect statements as not just incorrect, but confidently incorrect, a la Dunning Kruger.
See how confidently I made the exaggerative statement above? I don't necessarily mean it to be completely true, but I am making an argument. I think an assessment of confidence requires more than seeing no mollifying qualifiers like "I think" or "it might be". There's no verbal tone on the web.
(If it wasn't clear, I'm poking at the idea that we have numerous biases that prevent objective evaluation)
People think the high pay and the fancy titles* they're (often) given reflects their value or intellect*, even subconsciously, and they behave in such a manner.
*Sorry, I don't consider web programming (which comprises a majority of modern software development) "engineering"
*Many are some of the most intelligent people quite literally on Earth, or are otherwise exceptional.
If you’re told that you’re a superhuman, then why not think you can get away with it?
treating everything as a market or something to be solved with a market mechanism
People that have logic training such as lawyers and engineers even more so.