It may be the case you can put a new OS on the device without it being wiped, because the OS partition is separate from the user data partition, and probably unencrypted.

Anyway, a hypothetical alternative firmware couldn't just magically bypass the encryption. What it could have done, and I think this may only apply to older iPhone models as it's now handled by the Secure Enclave(?), is make it easier to brute-force the pincode (no lockout, less delay).