If GrapheneOS is not tightly sandboxing them, then chances there are that a capable operator can use whatever backdoor each driver offers, mainly the wifi adapter, the baseband modem and the Bluetooth adapter.
No matter what GrapheneOS developers have done.
Imagine the wifi driver being able to spoof on pin entry procedure.