I use the more conservative numbers partially to account a little for inflation (inflation-adjusted, the S&P returns over the last 30 years are more like 8x, so honestly I'm not even pessimistic enough with that number), but mostly because I am using the numbers as a way to manage my own expectations about investment growth over time.
If I invest $10,000 today for my retirement 30 years hence, I'll have $100,000 in 30 years; if I want to invest for 30 years and then live for 30 years off the income, I should plan to put away 1/10th of my desired 30-years-from-now income per year. Easy peasy. I shouldn't, say, expect to put away 1/4 of my paycheck for five years and retire to live off the dividend income in perpetuity (for most combinations of "income" and "expenses"); compound interest doesn't work that fast.
Regarding short-term risks ... I'll confess to buying the odd long-shot stock and lottery ticket, but frankly once you begin managing your risk it becomes a lot less exciting -- risk management dilutes the gains as well as the losses.