Go stack traces and the errors package
dave.cheney.net
dave.cheney.net
What's worse, is this solution doesn't solve random third-party packages you depend on all that much since they'll still use the utterly useless std-library error package or their own variation which won't necessarily be compatible.
The standard library's error package recording call stacks as this one does.
The standard library's error package supporting more rich information than just a string, but having both a message, a parent error (recursive data structure), a code-location (package / file / line number), and structured metadata available.
Rich tooling as rust has in the form of Either + macros to make it easy to do the right thing(tm)
... And I know most people won't like this, but exceptions would also solve this. Note all the above suggestions are viable without exceptions, but exceptions are a fine solution in their own right.
This really is just an area the standard library is utterly deficient in (much like logging), and the lack of macros, generics, and sum types makes elegant error handling much more difficult.
(Note: post edited to remove snark.)
P.D.: The downvote didn't come from me.
A big part of the style and simplicity comes from not needing to worry about what possible exceptions could come bubbling up from a library stack and under what conditions.
Even though people complain about the repetition of "if err !=nil" at least you know from the function declaration if an error is could occur.
(0) A correct result (and no error object)
(1) An error object (and no correct result)
Ruling out:
(0) Both a correct result and an error object
(1) Neither a correct result nor an error object
By “ruling out”, I mean “it's a compile error”, rather than “it's a pattern, and the community won't appreciate it if you deviate from it”.
Panics as normal control flow are just as unidiomatic in Rust as they are in Go, but at least:
(0) There's a decent (in fact, superior) alternative for error handling.
(1) You don't risk leaking resources. RAII has that covered.
Consider the case of io.Reader: https://golang.org/pkg/io/#Reader
Read always returns the number of bytes read, and if an error occurs after some bytes have been read (or at EOF) it returns that error as well.
It does feel a little dirty, but it's also efficient and pragmatic.
Sum types can do this as well. In OCaml syntax:
type result =
| Success of int (* bytes read = buffer length *)
| Error of int * err_info (* bytes read, error info *)
class type reader =
object
(* attempt to fill the buffer *)
method read : buffer -> result
end
The pragmatic thing to do is to make illegal states irrepresentable, so that the programmer doesn't have to worry about them. There's literally no benefit to losing conceptual clarity.
What I mean by conceptual clarity is knowing for sure, without looking at the documentation of each function, that it returns either an error or data for further processing, but never both.
That's not the case with your solution either. You're just packaging the same conceptual ambiguity in a slightly different way.
It is. A single call to the `read` method of a `reader` object returns either `Success` or `Error`, but never both. It's handled like this:
match my_reader#read my_buffer with
| Success(len) -> (* success logic *)
| Error(len,info) -> (* error-handling logic *)
There's no ambiguity whatsoever. The cases are separate, and you can't use the variables of one case when handling the other. There's still literally no reason to sacrifice conceptual clarity.
But your solution cannot fix the conceptual lack of clarity that follows from the decision of the API designer to return both error information and data for further processing from the same function call.
It's not a formal problem.
Why is it unclear? If the API supports using a reader after an error happens, the types have to reflect just that. It's not the kind of API I would design, but it's what you asked for.
> It's not a formal problem.
The inability to express your intention in a mechanically enforceable way is very much a formal problem.
A language that uses sum types (whithout any further guarantees) for error handling cannot formally enforce that invariant.
What I called pragmatic was that we may not want that sort of formal guarantee because we may sometimes want to trade conceptual clarity for efficiency.
Huh? The invariant is enforceable:
type result =
| Success of result_you_want
| Error of error_info
> trade conceptual clarity for efficiency.
Could you describe where exactly the efficiency gain comes from? If indirection is the problem, well, if anything, the Go approach requires more indirection.
No, what you are showing is how an API designer could enforce a particular API design decision. That's fine, but what I'm talking about is how a language designer could enforce the invariants of a particular error handling concept. I think that sum types alone are not sufficient to do that.
But I have to say I really don't like your tone. I'm not asking for one thing and then the exact opposite. I'm simply discussing pros and cons and I'm thinking about the consequences of different designs.
I'll make my mind up when I feel I know enough the subject to do so. If that offends you, then I suggest you talk to someone else.
Different APIs have different ways to handle errors. The language designer's job is to allow the programmer to express his intention in the most precise way possible.
That's exactly what Go's API does: it returns error information and "data for further processing" from the same function call.
Java has a feature called checked exceptions that ensure that you handle all possible exceptions in your program, although admittedly this feature is annoying. Another language called Nim[1], that is in fact a pretty good Go alternative, offers a similar feature[2]. But in Nim it's opt-in and the documentation includes information about each procedure and the exceptions it raises, this is shown by the `raises` pragma where `raises: []` means that no exception is raised. For example, you can easily see that `parseInt` may raise a `OverflowError` or a `ValueError` in the documentation[3] whereas `normalize` doesn't raise anything[4]. This information is inferred by the Nim compiler automatically.
In my opinion this is the best of both worlds, because when your program does crash, you get a nice stack trace that is just as informative as Python's stack traces and you always know the exceptions that each procedure may raise. Here is an example stack trace from a compiler crash:
Error: internal error: getUniqueType
Traceback (most recent call last)
nim.nim(97) nim
...
msgs.nim(957) internalError
msgs.nim(866) rawMessage
msgs.nim(863) rawMessage
msgs.nim(796) handleError
2 - http://nim-lang.org/docs/manual.html#effect-system-exception...
I think it's mostly annoying because checked exceptions were used badly in the early years of Java, so there are methods which throw loads of unrelated exceptions which have to be handled separately. Better design of exception hierarchies, and the availability of multi-catch, makes this quite a bit better.
Although it's recently got worse again because none of the shiny new stream stuff supports lambdas which throw checked exceptions. You have to either stick to unchecked exceptions and lose type safety, or probably better, rotate the exceptions sideways in type-space and use an Either:
https://github.com/poetix/ambivalence/blob/master/src/main/j...
I see that as a secondary issue, I feel the primary issue of checked exceptions is that handling and manipulating them is a terrible experience and even today I don't believe you can be generic over checked exceptions (though I could be wrong).
> Although it's recently got worse again because none of the shiny new stream stuff supports lambdas which throw checked exceptions.
It's not new though, the type-level facilities lambdas use simply don't allow for that. That's been an issue since generics were added, and even since the original release of Java.
Checked exceptions can only go through stack frames which specifically knows about them. And that, I think, is why they're such an awful experience and a failed experiment.
You can be somewhat generic over checked exceptions. This is okay:
@FunctionalInterface
interface RiskyGetter<T, E extends Exception> {
public T get() throws E;
}
public <T, E extends Exception> T get(RiskyGetter<T, E> getter) {
try {
return getter.get();
} catch (E e) { // <-- can't catch a type parameter
e.printStackTrace();
return null;
}
}
As a result, the streams stuff absolutely could have been written to handle exceptions. The standard Function interface could have been defined as throwing an exception, and the terminal operations could have thrown it on. Could, but i'm not saying should - the additional complexity of the interfaces would have been annoying, so the flexibility would not have been free.
`Either`s can be nested, put in lists, passed as arguments, and pretty much everything else you would expect from first-class values. Checked exceptions can't.
We are talking about Go. Why do you basically say "f. you" to people when people points at some obvious flaws in the language ? people complain mostly because they are using it, if they were not, they wouldn't. Unfortunately your behavior is something I've seen often in the Go community.
But.. I was responding a post that was pointing out many things that are known to not be present in the language. These features weren't forgotten or missed they were excluded when the language spec was being decided.
For some usecases. For which you can just import Cheney's package, some other or roll your own. Which is Go's point: simple by default.
For what I'm doing with Go I do not need nor want anything more than a string, I do not need a debugger so I don't use Delve, I don't need logging more complex than what the std libs interface offers... So why should I carry all that sort of baggage by default? If you need that stuff, feel free to import the adequate package or use a language that fits your purpose better. IMO Go is better treated as a DSL than a general purpose language, although I think the same of most languages.
There is no other language that fits my purpose better. Does that have to mean Go is perfect? That I have to stop complaining ... even if it means no change will come about? Even if you think it may be perfect but I don't?
I think Go makes a very nice general purpose language. Better than the others I've tried: C#, Java, D, Rust, Dart, Python, Elixir, and more. All have critical flaws that make them worse for my purposes. If another better language comes along I'll switch. In the meantime I'll continue to use it but will complain about the things I wish I could change but don't have the ability to change myself.
>> ...although I think the same of _most_ languages.
Please elaborate on the non-mosts. I likely have missed many in my searches.
My experience is close from yours. Could you briefly describe the critical flaws of each language you listed, that made you choose Go instead? I'm geniously curious.
Elixir: The BEAM runtime, if I'm going to be fair to C# and Java. Elixir is the most elegant language I've seen in years. I really wanted to like it. I purchased all the available books and really dove in. But then I stumbled on the soft spot of number crunching essentially being left out of the language at the lowest level! That's a tough one to get over as a couple applications I have do lots of financial "what if" calculations. Also Phoenix has an annoying list of dependencies (Node???) and a complex configuration compared to Go.
D: I wanted this to be my primary language after reading Andrei's book when it first came out. Unfortunately D wasn't there yet (this was during the shift from D1 to D2). And after D2 solidified as "D" it has has been moving towards memory layout, CTFE, removing garbage collection, and other lower level stuff that I just don't care about. All of my applications are at a business level and dealing with memory and performance optimization is just not something I am ever concerned about.
Rust: Rust sits at an even lower level than D. I really like the higher level ideas and syntax but I absolutely don't want to think about ownership issues. If that part of the language were removed Rust would be pretty ideal. But then it wouldn't be Rust. Manually controlling the ownership process is a core part of the language and would be pretty hard to change.
Python: Bad deployment situation on Windows. But also I've decided I really prefer the compile time type checking. And while I notice the speed issue I can't really say it's a problem for the applications I code. But I do like the blazing speed of Go and the other compile to the machine level languages.
Dart: Hits on most every item on my checklist. But again, a bad deployment situation. And Dart's focus has shifted towards the JavaScript side and away from a general application language and server side language.
Nim: I'll add Nim because it shows so much promise. I've followed Nim for years, support them on Bounty Source, pre-ordered the book, and keep up with progress. However it's got probably close to a year before 1.0 and probably three or four years before it is as production ready as Go is now. But all in all it's shaping up to really hit the sweet spot for business level apps at a compile to binary level.
Honestly, Go has so much going for it that the items I find missing just aren't nearly enough to stop using it. But, for me, it could be so much better with just a half dozen additions. It's frustrating to see it so stagnant after just being born.
Ownership isn't a “low-level” issue. It arises in every program that manipulates ephemeral resources (like network connections), and you have to reason about it, whether the type checker forces you to or not.
> Manually controlling the ownership process is a core part of the language and would be pretty hard to change.
It's actually pretty automatic. You only need to implement each destructor once. Using `finally` or `defer` is what is needlessly manual.
Ideally I'd like the compiler and compiler syntax to fade away as much as possible while coding my applications. Only the domain syntax should remain: the syntax that allows me to turn my thoughts into working code free of any distractions to satisfy the compiler, error handling, etc.
I wasn't talking about memory management, for which garbage collection is indeed an adequate solution. I'm talking about other resources, which aren't as plentiful as memory (e.g., file descriptors) and thus must be reclaimed in an eager and deterministic fashion.
That's interesting for me to read your comment because my requirements are similar to yours (mostly business level apps with some number crunching), and my experience with each of the cited languages is similar too.
Another reason that motivated the switch from Java and Python to Go in my team, was the concurrency and the ability to manage a large number of asynchronous IO while keeping synchronous code.
-- Continuously bring up that something is wrong.
-- Get them to agree that their viewpoint is good but not perfect.
-- Get them to agree that a change wouldn't be bad.
-- Get them to admit that a change might be good.
-- etc.
We're just now finishing the first step the are into the second. We keep going. Fight on soldiers!
There is no error package, perhaps you mean the errors package. But it doesn't matter how you create your error in Go as long as you assert for behavior, because errors are interfaces. That is the point really.
If you import a package that uses Dave's error package, in general, you don't need to import Dave's package. That is the point of interfaces, to decouple things.
Of course (s)he does. Why even point that out...?
Firstly, "to reliably get error stacks" doesn't mean "to get stack traces"; it means "to get stack traces attached to errors". You do indeed need Dave's package, or some equivalent, for that; it's not something the standard way of creating errors gives you.
Secondly, "this solution doesn't solve random third-party packages you depend on all that much since they'll still use the utterly useless std-library error package" doesn't refer to the situation where "you import a package that uses Dave's error package", but the converse - if you import a package that uses the standard way of creating errors, you won't get stack traces on those errors. That's true, and it's why Dave's package provides the Wrap function as a partial mitigation.
There is no general and reliable way to handle specific types of errors.
And there is no way to add instance specific information to errors in a later version of a package, because that would require using an error type instead of (or in addition to) the existing well known error value, which breaks callers.
Yes you can. Errors are interfaces, not values. As long as you define an Errror() method your struct behaves as an error, but it can have anything a struct has, such as specific information.
You should read Dave's previous presentation (http://dave.cheney.net/paste/gocon-spring-2016.pdf) to understand a bit more about what's already possible today and why Dave's package makes it easier.
var Error = ...
func Get(url string) (result, error) {
if ... {
return Error
}
}
r, err := net.Get("http://example.com")
if err == net.Error {
//handle this particular error
}
So if you want to know if this error is temporary, you should have this, inside the net package:
func IsTemporary(err error) bool {
// specific parsing
}
func ErrorCode(err error) int {
// a type assertion, which is fine because we're inside the net package
v, ok := err.(ErrorWithStatusCode)
if ok {
return v.ErrorCode
}
}
Basically your package should give all tools so that callers can take a decision without knowing about internals of the package. Knowing the exact error code is probably not what a caller wants; what they probably want to know is whether the error is temporary or fatal, if they can retry with a chance of success, if they can take counter-measures and if they can't alert the user.
The standard library is already littered with error values defined on the package level. Many third party packages emulate that approach.
Once you have done that in package, there is no way to change it. You cannot add function call specific data to an error after the fact. There is no way to implement an IsTemporary(error) function once you have clients that compare any returned error to a package level singleton using ==.
>Knowing the exact error code is probably not what a caller wants
I had that need many times. For instance, I needed to get the SQLSTATE from database errors. There is a huge number of SQLSTATE codes and interpreting/classifying them can't be the job of a data access package or even a DBMS specific driver library.
Also, how would a networking package know what I consider temporary? Some protocols like HTTP specify some errors to be temporary, but that may not coincide with what I'm willing to retry or not retry.
You can also create a type or a struct that implements error.
Example from stdlib:
``` // PathError records an error and the operation and file path that caused it. type PathError struct { Op string Path string Err error }
func (e *PathError) Error() string { return e.Op + " " + e.Path + ": " + e.Err.Error() } ```
Dave Cheney's other blog post/talk goes into various ways to handle errors in Go here: http://dave.cheney.net/2016/04/27/dont-just-check-errors-han...
Once you have decided to use a package level error value (sentinel error as you say), there is no way to change that.
As there is a huge number of such error values in the standard library and also a substantial number of other types of errors like PathError, "there is no general and reliable way to handle specific types of errors".
What you can't do is add call specific information to errors returned from a particular function if that function returned a package level singleton value before.
I see that other people have made the argument in support of Dave Cheney's approach already so I won't repeat it.
The language problem is that laziness is enabled. The real, practical problem is that programmers are lazy.
It would be nice if at some point the standard library would be scrubbed of string errors. But for the most part, the standard libraries errors that are sane to disambiguate are disambiguated, either with error types or helper functions.
(a) err == pkg.Err
(b) pkg.IsXYZError(err)
(c) switch err := err.(type)
If (c) is discouraged, as some have claimed on this thread, then how are we supposed to access type specific data?
Also: cards on the table: if there's no straightforward way to respond distinctively to different errors, I don't think error strings are really all that bad. I think it's reasonable to look at untyped errors as "this library function returns only one kind of error: it didn't work". A lot of legitimate interfaces are like that. Why spend extra effort typing their returns?
So let's simply return errors.New("...") or a private package level error value.
But public package level singletons are a completely unnecessary burden on API versioning with no upside at all.
Why on earth would I ever want to make a lasting commitment to never attach any data or call specific message to errors in the future?
1. Assert error behavior not type or message, i.e. the error has a specific method. NOT `type HTTPError struct {Message string;Code int;}` but instead `type HTTPError interface {error;HTTPCode() int;}`. This decouples the location and specific implementation of the error type from the extra information you're adding to it.
(Corollary: Interfaces are Go's only real way to create custom generics. You'll be much happier if you can map your problem space onto interfaces.)
2. Number 1 doesn't help when you wrap errors, because the wrapping error hides any methods on the original. Dave's errors package helps with errors.Cause() that exposes the original error so you can apply #1 effectively.
[0]: http://dave.cheney.net/2016/04/27/dont-just-check-errors-han...
But the standard library uses sentinel errors everywhere and third party packages often do the same. That's what creates the inconsistent and messy status quo of handling different types of errors. It's also not going to change because sentinel errors can never be changed into anything different without breaking clients of the API. No call specific message or other data will ever be added to these standard error values.
Dave's post and my entire comment directly address this: Add a `SQLCode() int` method to the custom error, return `error` type and users can assert `codeErr, ok := err.(interface{SQLCode() int}); code := codeErr.SQLCode();` somewhere up the stack to get it.
But yes, the standard library's use of sentinel errors is unfortunate and can't be fixed.
You're, right. Sorry for that.
So almost all packages either return a base error, or a custom error type that adheres to the error interface.
>Some errors can only be identified by value and others can only be identified by type
Let's say I'm reading a csv file, and I need to check whether I'm done reading the file - I'd do something like so:
if _, err := csvReader.Read(); err == io.ErrEOF {
quit()
}
if c_err, ok := err.(LibError); ok {
c_err.Code
}
Go errors are not exceptions.
If the latter: fair enough, but it's not as if that's a common feature of other mainstream languages.
The thing that really sells it to me is the being able to wrap the error when necessary with errors.Wrap and find the underlying cause with errors.Cause. I've yet to experiment with the new formatting `%+v` but I can see that coming in useful too.
There have been earlier attempts at something similar (eg https://github.com/juju/errors) but none with the same clarity of thought.
So thanks for a great package Dave and for taking the time to whittle it down into the simplest, most elegant thing.
Honestly I don't know why stack traces aren't formatted for quick parsing. Many thanks to maruel [1] for the inspiration.
My package in it's entirety is:
package errors
const ErrUnknown = 0
type RESTError struct {
HTTPStatus int `json:"-"`
Code int `json:"code,omitempty"`
Message string `json:"error"`
Err error `json:"-"`
}
func (e *RESTError) Error() string {
return e.Message
}
func New(status int, code int, message string) *RESTError {
return &RESTError{HTTPStatus: status, Code: code, Message: message}
}
When those errors finally get back to my web handlers, a standard renderError wraps all errors:
func renderError(w http.ResponseWriter, err error) int {
if e, ok := err.(*errors.RESTError); ok {
return render.JSON(w, e.HTTPStatus, e)
}
e = errors.New(http.StatusInternalServerError, errors.ErrUnknown, err.Error())
return render.JSON(w, e.HTTPStatus, e)
}
It effectively acts as a map between internal errors, and external communication of errors, and allows the place where the error occurred to say "this HTTP status". And because the Err is never stringified I am comforted that internal sensitive data (this is a billing system and unsanitised data should never be leaked even via error messages) does not get leaked.
That's my goal, to handle the error in the way that allows it to best communicate to:
1. HTTP clients via the status
2. Developers via an error code and sanitised message (which they can use to communicate to users)
3. SREs via detail in the secure logs
I found the native errors nice, but not the best for dealing with my 3 audiences.
type AppError struct { Code int Message string InternalMessage string Time time.Time HttpCode int }
All my errors are handled in a http handler function. Before that these errors are produced in any functions called by my handler functions.
I concatenate in to the InternalMessage var any error message I want to log, basically anything I want to show an admin. The message tag will have anything I want to show a user. Normally separated by a semicolon.
Because of this I can give good errors to my users, and I have enough information when something goes wrong.
The only negative might be that, the internal error logs are a bit verbose.
the code above should be
type AppError struct {
Code int
Message string
InternalMessage string
Time time.Time
HttpCode int
}So http 4000 is bad request , but in the business logic terms
https://github.com/go-kit/kit/blob/master/examples/shipping/...
and more recently https://github.com/go-kit/kit/blob/master/examples/addsvc/tr...
This technique and the OP show the power of "programming errors" in Go.
If I'm calling `foo.Bar()` directly, I already have a direct dependency on package `foo`, so what difference does it make to have another one to get `foo.BarError`?
If I'm calling `foo.Bar()` indirectly, then it's true that checking `foo.ErrorType` introduces a novel dependency upon `foo`, but it's equally true that:
a) my direct dependency is frequently going to check and wrap that error in some other type anyway, and
b) I can still use the interface trick that Dave describes to avoid a dependency on `foo`.
I don't see why forcing callers to declare helper interfaces every single time they want to inspect an error is preferable to them only having to do that as a rare workaround.
The difference is that the latter has to be translated to be useful (even when your user understands english).