It's really not. Most of the software you use every day was developed by very average software developers. The vast majority of the work software developers do isn't terribly novel or particularly difficult. And you know what? They do just fine. Things work well. Systems stay on line. Companies make money.
I've seen too many "bad" teams produce perfectly functional software over the years to think that this gap exists anywhere but in Joel Spolsky essays.
I think comparing carpenters to software engineers does a disservice to carpenters. Software developers to me are more like dry-wallers. A really good one can do it faster, but the end result mostly always looks the same.
The difference between an average and a great carpenter, however, is immediately obvious. There is real craftsmanship there, and you can clearly see the difference.
It isn't until you get to the edges of software development that those same differences become clear. When you're dealing with very high scale, very aggressive performance metrics, or very difficult mathematical concepts you see the difference between average and great developers. But those are less than a fraction of 1% of software projects I'd wager.
* I should note: I'm a software developer.
The key factor that you're probably not thinking about is time. A good dev doesn't think faster or have blazing fingers that fly across the keyboard, a good dev is constantly setting him/herself up to shave time off of future tasks, and those investments are constantly paying dividends.
Those "bad" teams can definitely produce software that runs, but they do it 10x slower than a "good" team could.
What takes a bad team of four six months to make takes a good team of two only 5 or 6 weeks.
Time is the key. Writing maintainable code is the investment a good developer makes because it pays off by increasing speed of interacting with that code.
I should note that I’m a software dev and a family member is a general contractor
If you have great developers, there will be a single straight wire from the switch to the lamp.
The end result for the user is the same though, until the mess of wire eventually overheats and burns down your house.
Projects that don't fail, usually have to be rescued from some crazy doomed plans fairly regularly.
You seem to have found a calmer, saner corner of the industry.
For example, the people at Apple who came with blocks and Grand Central Dispatch did an amazing job, but the difference between them and the average software developer is like 100x. A great software developer can write something that other developers can build on, which makes them more productive.
Conversely, the people who came up with Java, they figured out how to turn a great programmer into an average one, and make them want to quit computers altogether, but the below average programmers are now also average.
While I do feel that folks grossly underestimate the skills that the best carpenters acquire over time, the software ecosystem generally is unique is that it's vastly large and complex. By any measure (switch points, surface area, person-years, etc.), the software we've all built in the past 70-ish years is the most complex thing ever implemented by human-kind.
Long ago it became far too large for any one person to "master" to the same degree an excellent carpenter might master his or her skills.
And the market agrees with the point of view, and values the economic output of the "best" software developers (at scale, not just one data point) at a far greater multiple over the "ok" ones, relative to other skills and trades.
One may not like this analysis (and my associated statement), but it is the state of the world.
Any profession to which someone can devote their life to (incl. carpentry or software development) is going to have gaps in skills between "ok", "good", and "great". Pejorative attitudes which measure something like software development as "harder" than carpentry are not beneficial and simply dilute the discourse.
Especially as many software devs are becoming more like "code plumbers," if anything the skill gap may be bigger in carpentry in the future.