Weather is governed by impersonal physical forces which can be described by mathematics. Software is built by groups of people who most defitely cannot be modeled as mathematical entities. Weather is a bad analogy here.
> "The equities market is an extremely complex system, yet we use market forecasts all the time. What's different between equities market and software is that everyone knows that equities market is extremely complex, and we account for that when we think about the price of an equity in the future. ("Maybe I'll diversify my investments.")"
Those market foreasts are often wrong. Even a company's own forecasts are frequently wrong, despite having the optimal perspective from which to make those forecasts.
Furthermore, diversifying investments is easy to do because the price mechanism makes those investments fungible (in a liquid market, anyway). This is not the case with software. Each piece of software is unique. You may "diversify" your software by investing in multiple projects, but ultimately you have a business need for a piece of software which does task X. "Diversifying" by funding software that does tasks Y and Z will not help you if the software "investment" in task X fails. "Diversifying" in three software projects, all intended to do task X, will roughly triple your costs for doing X. Again, equities are a bad analogy.
Software development is a process which can't be easily summarized by analogies to existing systems. In fact, the term "software development" can describe many different systems.
The real reason schedule estimation in software development is so hard is because, regardless of which system of software development is used, the problem of simultaneously optimizing to maximise a desired quality while minimizing the cost/schedule is inherently hard in a mathematical sense. Changes which promote one oppose the other, and the sensitivity of a software development system to unknown and unpredictable future changes cannot, obviously, be predicted in advance.
Your point about hedging one's bets is valid, but there are costs to doing so which are not as simple as carrying an umbrella or splitting up a pile of money across multiple equities.