I'm not a mobile phone security expert but my feeling is that in the case of GrapheneOS - which target is probably high-profile people at risk of state actors et similia attacks - a zero-day in the closed source firmware from Qualcomm will probably screw you anyway.
I understand that you are anyway reducing the attack surface (now they need to target the modem firmware specifically), I understand the concept of security in depth and I also understand that by using GrapheneOS you are already placing mitigations for many other known and unknown attack vectors. But still...