Following this hint turns out I had left out the `replace={true}` prop on react router's Navigate component. Thank you for taking the time to point me in the right direction!