What are the dangers of graphite lubricant?

I lubricate my keyholes with it but there is always some that gets left on the key, and I tend to fiddle with my keys at my desk sometimes.

Occasionally I’ll get some of the graphite on my hands, or possibly even breathe it in I’m not sure. Should I be more careful with this stuff?