Not all nanomaterials are dangerous (just as varying sizes of dust are not), but many new materials may adopt unnatural properties that our body doesn't know how to deal with. So there is still handwavyness to it all.
You're right that Graphene is ok, though.
A lot of nanomaterials have the potential for creating dust in this size, including ones made of pure carbon.
Is lunar surface material a nanomaterial? It's certainly nasty stuff and it gets everywhere. It's regarded as one of the primary risks in lunar colonization. Does that make it a nanomaterial?
Also it has been the case with every round of technology that there are unexpected risks that turn out to be important. It would be a surprise if nanomaterials failed to bear as yet unknown risks. Particularly once you start doping them with a variety of trace elements to get the properties that you want.