Would your abilities feel stretched by being a cashier? How about a janitor? Waiter?
These jobs are what "work" actually is for most people. I think most people could go without doing those things.
Intellectual stimulation, or goals that stretch one's abilities, don't have to come from work (and for most people those things and "work" are hardly related at all). Therefore, saying we (as humans) "need" work is generally incorrect (at least for the reasons you stated) since most people don't even get what you say we "need" from work anyway.