There's a new-ish book, "Secure by Design", that makes this point -- and shows ways for ordinary devs to design for security without explicitly thinking about security.
I really like this book so far (it's still in the early access stage of writing).