It's actually preferrable in many of those "infallible" cases to panic rather than handling the error since it usually indicates that your application is in unknown territory and there is no "safe" way to handle it.
If you're writing cryptographic code, panicking for every issue is not good, that's a DoS vector that could be easily exploited. It's safer to handle the error in-band and let the application higher up decide if it should panic, error or continue.