A lot of the drugs with strong effects necessarily have long term consequences from repeated use owing to their mechanisms of action.
You risk making yourself 'dumber' over time with prolonged use. Especially, with the amphetamine based crap that US doctors seem okay to use for medical purposes despite most of the developed world not doing so, and using safer alternatives.
The major concern with Modafinil is that we just don't know what the long term side effects are. That's the risk you take.
Long term effects are tolerance, and.. not much else. It has to be cycled regularly. If you abuse it or fail to socialize/have fun enough it'll end up making you feel like a zombie. It's been in regular use for a decade and a half, I haven't heard of any serious issues.
The main risks are the first time you take it. If you're allergic you can have some very serious reactions. In addition, if you're in a high-risk group (low immune system, etc.), you want to avoid the slight risk of SJS/TEN is can cause. SJS/TEN is probably the most terrifying risk, although it's extremely rare, and hundreds of other drugs can cause it too.
Each of these substances are amazing, but not without some danger.