There is a tradeoff... with young kids and a full-time job the time I can spend buying or preparing meals is pretty limited. I don't fully understand the magnitude of the risks of, say, eating too much vs. eating too much pesticide.
I'm also kind of irrationally mad at organizations like the Environmental Working Group; as far as I can tell their "dirty dozen list" comes from taking public domain FDA data about toxins in food, dividing the guideline safe amounts by about 10 arbitrarily, and publishing that the public numbers are higher than their (apparently made up?) lower safety thresholds. I just don't understand the science here -- how much extra death are they saying I'm incurring by eating non organic produce and what are the error bars?