One of the only big downsides I've noticed with GrapheneOS is that several banking apps don't work with it at all thanks to being tied to Google's verification ecosystem.
Luckily I have hardware 2FA keys from my bank so I can authenticate using that. It also slightly decreases the suck-factor from whenever the phone decides to fly off down a drain. This may not be the case for you, so do your research on what you need for daily living.