Ideally, you can constrain the set of inputs to only valid ones by leveraging types. But if that's not possible and a truly invalid input is passed, then you should panic. At least that's the mental model that Rust is going with.
You do lose out on the ability to "catch" programming errors in subcomponents of your program. For example, it's extremely useful to catch exceptions related to programming errors for called code in response to a web request, and return a 500 in those cases. One could imagine a "try" "catch" for panics.
The thing is, it takes a lot of discipline by authors to not riddle their code with panics/exceptions when the language provides a try/catch mechanism (see C# and Java), even when a sensible error as value could be returned. So Rust opts to not introduce the footgun and extra complexity, at the expense of ungraceful handling of programming errors.