Imagine what Steve Jobs could have done if he hadn't gotten into drugs! I bet Barack Obama could have gone so much further in life if he hadn't smoked weed.
Smoking pot does effect the drive of many people, whether you would like to believe it or not. Some people are better at functioning under the influence than others.
Two examples of people that dabbled in drugs does not make your case.
I could give you two examples of couples that have sex without protection every day and have never gotten pregnant. Does this mean everyone can do it with the same outcome?