It's just that the distance above ground at which the buoyancy changes sign is much, much higher :)
I guess it might not be a hazard at all. Firstly, they wouldn't be traveling at orbital velocities, and secondly, since the premise is that they are the same density as the air, hitting a float stone at Mach N might not be any worse than hitting the air at Mach N.
Dive into a swimming pool filled with water. Safe, harmless fun!
Dive into a pool filled with solid ice. Ice is 10% less dense, so ten percent more safe and harmless and fun, right?
Density is not the most important physical quality here.
Although it would have to be over 66 cubic meters for an 80kg person in 20°C air. If it is spherical, it would be c.a. 5 m in diameter.
So, the volume of each needed to lift 80kg (which would have to include the person and whatever structure to contain the material), is 80/(1.2 - density of material) m3. So:
Helium: 78 m3, or sphere of diameter 5.3 m
Graphene aerogel: 77 m3, or sphere of diameter 5.3 m
Hydrogen: 72 m3, or sphere of diameter 5.1 m
Crazy things ...
Of course, nothing mentioned here makes any sense since the material is not rigid.