Software Engineering is still in the stage of early industrialism - when James Watt built his steam engine, or Carl Benz built his car, they did not have specification documents. They tinkered and made it work, moving around assemblies until everything fit, then improved on the design. Essentially, they were doing some form of 'agile': small teams (to the point of only one), no specifications, high turnover. Today, this approach is unthinkable in mechanical engineering, because bad designs are lethal.
The software engineering environment is special because when it started out, everyone thought of it as just another branch of engineering, so they applied standard industrial processes onto it - and that did not work well, what worked for building hydraulic presses did not translate well to writing software. SE needed to "learn" how to tinker and experiment on mid-sized projects first, and now they are in an early-industrial-age phase (and call it 'agile').
Eventually, SE will return to something more organized, something more reliable - after all, we are learning that bad design choices can be lethal. We are seeing the first steps today, trying to capture agility with frameworks, with dedicated test methodologies, ... Naturally, this will cause frustration with the tinkerers, and it will take time. And while some aspects will resemble mechanical engineering processes, some things will be completely new and untranslatable to other engineering disciplines.