I bought a few years ago and did the same calculation. Despite prices up 30%, my down payment would have produced higher returns if I had invested it.
Plus stocks are almost immediately liquid and have almost no transaction and no maintenance costs and no annual property tax.