It isn't the CPU that's saving all registers here, it's the interrupt handler code, which, if the author wanted to reduce latency, he wasn't forced to save all registers.
Yup.. If author's working with a RISC-V softcore, there are several options to reduce interrupt latency:
# If softcore saves many registers as a hardware feature: adapt it to not do so
# Save (and use) only a few registers in interrupt handler
# Reserve some registers to be nuked by the interrupt handler, and don't use those in application code
# Use another softcore with faster interrupt response
# Move to a part with cpu as hard silicon (with sane interrupt handling)