Consider that using the same settings with different parts might lead to subtle instability, not a full on-boot crash.
DDR5 tops out somewhere around 8 Gigahertz clock speed, which means the data signals travel 3 centimeters before the next clock signal arrives. At those time scales it means you're going to have to align those data signals down to the micrometer if you want them to be properly received.
You could do an experiment. Remove some of the RAM modules and see if the 20 second boot time decreases.
If you took out a RAM stick and inserted it into the same slot, the conditions (due to dust, moisture, difference in seating) could be enough to throw off the reliability of the training data. So at minimum it has to verify the settings to be correct.
[1]: https://www.systemverilog.io/ddr4-initialization-and-calibra...