Although it is not exactly proven, one can assume that asbestos' toxicity is related not to it's chemical properties but to physical geometry of particles of asbestos dust which seem similar to dust that you would get from carbon nanotubes. On the other hand, there is quite significant question whether this actually is problem (even in asbestos case) given right uses and technologies.

I could probably cite some examples of molecules where with a certain molecular structure/combination it's harmless, then you add just one more Hydrogen or Nitrogen, or whatever, and it becomes an acid.

Also consider water aka H2O. Have a sip of it and your mouth feels better. Have several cups and you're refreshed and body replenished for the day. Have 50 gallons go down your pipes and you are dead. Same molecule, phase, and microstructure. But water is extremely old and long studied and literally falling out of the sky for millions of years while humans on the planet. Not so, graphene.

Believe me, I think it's cool and sexy and do hope it enables all kinds of breakthroughs, I do. I'm just skeptical about whether it won't unleash horrible health problems down the road, due to incompetent and/or corrupt testing/approval bureaucracy, especially in the USA.

Graphene is a 2D material, like a sheet of paper. A carbon nanotube is more of a 1D material, like a pencil. Carbon nanotubes have (extremely) sharp points. A single wall carbon nanotube has a diameter of ±1.4nm. That is a lot smaller than the size of a cell. The danger of nanotubes in living organisms is that they can puncture cells. A graphene sheet cannot do that. That’s why nanotubes are potentially more harmful than graphene.