Coming from Python, I was of the same opinion at first. A significant amount of error handling lines is what I noticed first when looking at Go code[0], too.
However, I was interested to learn that Google style guide for C++ recommends not using exceptions. I don't know C++ (shame), but at least some of their reasoning is applicable to Python as well:
For example, when you start raising an exception in a Python function, you have to check its callers, whether they (or their callers) handle it. And vice-versa—when a Go function has “expected” error in its signature as return value, it's very obvious what you need to handle when you're the caller.
“Expected” error as return value also encourages single responsibility principle, I suppose. E.g., for a function that decodes JSON, bad syntax is “expected” error, the rest is a reason to panic().
I'm not sure if I get your example with refactoring, but it may be a case when you use panic() in inner function, recover() in outer, and return an error as usual.
That said, I've never actually used Go yet, so it's just theorizing.
[0] But then my Python code that (to me) looks more or less “solid” appears to contain a comparable amount of error handling lines, so not sure if it's a good metric.