My issue is mostly implementing both ways of bubbling up an error to somewhere it can be handled. I think having either error return values or exceptions is preferable to having both. I don't think exceptions are perfect, but if panic() absolutely has to exist then I'd rather have an entirely exception-based language than a language that uses both systems simultaneously.
E.g. if I write a function that accesses an element of an array without bounds-checking, it could panic and I have to handle that exception. Bounds-checking basically just becomes finding things that would throw exceptions and converting them to errors so we can pretend that exceptions don't exist.