The problem with your doctor's recommendation is that doctors recommend colonoscopy based on age, not whether you have any factors that might indicate colon cancer. I have no cancer of any kind anywhere in my family tree and they are always on me to get one.
From the article, a doctor who still believes in colonoscopy for everyone: “The first message is that screening saves lives [Ed: against this study's data] and prevents cancer. If we could have a chance to start everyone at age 45, I’d like that."