I find that directly propagating errors upwards is an antipattern. In many/most cases, the caller has less information than you do for how to deal with the error. (Your library, after all, is an abstraction, of which such an error is an implementation detail.) It also prevents you from returning your own error condition which isn't one that the downstack error type can represent.
It may be that you do in fact have to throw your hands up, and propagate _something_ to the caller; in those cases, I find a Box<dyn Error> or similar better maintains the abstraction boundary.