The theory is that agriculture let populations better survive bad years by grain stockpiling so they could outcompete the hunter gatheres.
In general, nothing says that better average quality of life necessarily makes you out-compete other populations.