Is there something special about software with the level of introspection workers in this industry are expected to do, which cannot (or shouldn’t) be done elsewhere?It's not so much software, more engineering in general. You do find rigorous self-review and continuous process improvement in many areas of engineering (more commonly called quality control), Toyota being the obvious example.
However, you are right that most industries aren't like that. Compared to engineering most industries are very 'soft' in that correct results don't really matter or can't even be defined.
Take industries like marketing, retail, law, management consultancy. Look at the output of your average office worker in these fields. Most likely their output is in the form of Word documents, PowerPoints, maybe some Excel. Is the last ad campaign your firm ran "correct"? Did the management consultant who breezed around your corridors earlier give advice "efficiently"? Did the lawyer who botched your paperwork ever face any consequences for that or did they just go, oops, and do it again?
The vast majority of workers work in environments in which the connection between input and output isn't very clear, and in which being wrong is less important than having the right relationships with people. In such an environment a retrospective would not only be pointless but actively harmful:
- A retro inevitably involves frank discussion of people's work, and whether it was acceptable or not. Most people cannot separate criticism of their work from criticism of themselves.
- Therefore having your work be openly criticised makes you likely to dislike the person criticising you.
- So people may simply turn around and blame their colleagues or management for their own failings instead.
- Therefore the idea of frankly discussing a team's failures in ANY kind of setting, whether it's called a "retro" or otherwise, is largely anathema in many lines of work.
In engineering professions what we do must work and it must be correct, and it's very clear to everyone if we make mistakes. Therefore a culture develops in which people can separate criticism of work from criticism of themselves, in which people aren't usually called out specifically for their errors, and in which systems thinking dominates. The answer to "why did this project suck" isn't "because so-and-so couldn't make their mind up" which is the sort of answer you're liable to get from non-engineers. It's "our planned process didn't correctly refine our requirements from the start". Totally different mentality.