See pg.1259 in the datasheet, the ARM_DISABLE fuse does exactly that. Putting that in was a smart move because it means they can always make a RISC-V-only variant of the chip without having to tape out new silicon - they just have to blow that fuse at the factory.