We ended up switching to Django, where just an traditional http only secure cookie which we "proxied" from Next.js to the frontend and back was enough and maybe even more secure than all the mangling we had to do with the refresh tokens, etc.
I really wish this trend of fancy authentication mechanisms goes away. It doesn't take in account how easy they make for people to mess up on their implementation/usage of it when things are too complex.