Depends on the use case. I'd say that for many, many cases, all errors should lead to the same result in the application, but with an error-specific informative message. Even if you want to have special handling for errors such as 4xx that might mean something is permanently wrong, getting a 500 is the same as experiencing a network error (except in a client side app where the latter might mean you're not connected to the Internet and should try again, but then you should use yet another API to determine when to connect), and getting a 200 with the wrong content-type has an excellent chance of meaning the same thing as a network error, in the form of a captive portal. To this extent, APIs with many separate error paths that encourage the user to write their own nonstandard error messages are harmful, because they will probably do worse explaining things than with a common method of triage.
Of course there are other cases where the difference matters. It just... doesn't seem very "natural and idiomatic" to me, that's all.