You should note that large parts of the world's population have never consumed dairy throughout their history (except as infants, of course), and are still doing very well today (China being the largest such group). I don't think this is a good line of argument.