I hear you. The issue is, to me, that because interest are so low (and appear to continue to get lower) this has caused the list prices of homes to increase in proportion. Because of this, a 20% downpayment has become even more difficult to achieve for more people.
To be sure, the down payment doesn't protect the buyer at all. It simply exists to protect the lender in the case of an early default. They can still sell the foreclosed home, possibly under market or in a depressed market, and be made whole since they only lent 80% of the original price.
In fact, the average Joe would be better off in today's climate by putting 0 down and using what they would have put down into different financial instruments that will likely out perform the 3.x% interest rate they are likely receiving, considering they wouldn't have other penalties like PMI.
The financial crisis had a lot of inputs and bad banking was a part. That is, banks giving loans to those who were not financially able. However, a big part of this was due to government interference in the free market pushing President Bush's "ownership economy" policy which forced banks to give loans to people they probably wouldn't have otherwise. And then the whole thing went to hell.
A 0% down loan to a vetted individual with sufficient net worth wouldn't be a bad thing. I do believe there is a calculus we are missing between interest rates, inflation rates, and what a viable down payment should be.