It's not really taxes, although the fees for filing corporate documents. The franchise taxes are lower, but the numbers for large companies are so tiny it's inconsequential ($800 in California compared to $150 in Delaware).

My understanding is that Delaware has strong corporate law with well defined protections for corporations and lots of case law to back it up from their Chancery Court. The huge history of case law is really what makes the big deal from what I've read because it makes contracts enforced there (which all shareholder contracts would be) much more predictable in terms of what's allowed.