Measuring execution performance of C++ exceptions vs. error codes | Hacker News Reader

Measuring execution performance of C++ exceptions vs. error codes

nibblestew.blogspot.com

78 pointsBuuQu9hu70 comments

IMO the reason not to use them isn't performance or size (although those were legitimate factors in the past AFAIK) -- rather that they make control flow very difficult to reason about. They're worse than a go-to statement, they're a comes-from statement. They can pop out of anywhere and be handled anywhere up the stack. They always seem like a good idea at the time, and end up biting you in the butt.

Not to mention, that's old-school error handling, Result<T,E> types are what the kids seem into these days.

Curious how destructors will affect the numbers.

That result object is implemented in the same manner to an error code: you are burning cycles and registers on something that usually doesn't come up... all so you can clutter your code with what effectively amounts to a manual implementation of exception handling :/.

That's fair, though in my opinion it's more important to be clear and explicit in what's happening both for your own sake and for your future co-workers so they can understand what's happening and why than worrying about some minor (per the article) performance issues.

That's what exceptions are for. They are a tool that _can_ be used to cut down on noise in code. Being clear and explicit about everything will leave your code looking like an implementation of the STL. The art of programming is partly about achieving a good signal-to-noise ratio.

From a reasoning perspective throwing an exception is the same as returning from a function. If you want to reason about code which potentially throws an exception you have to keep track of a separate exception post condition.

If you compile exception handling to separate return addresses this becomes rather obvious. In this case your control flow also stays reducible, which is commonly taken to mean "simple to reason about", and is very different from the kind of control flow that you can introduce using goto statements.

I think the main problems with exception handling are cultural and psychological. Culturally we don't insist on proper documentation for exceptions. At best you'll see documentation for the functions throwing exceptions directly, but almost never for the functions using them a few levels up the stack... The situation is even worse with higher order functions for which people almost never document the exception behavior. Psychologically most people seem to ignore all but the most obvious exceptional cases. We really have to force ourselves to notice everything that could go wrong and to take a step back and rethink our designs if this question doesn't have a simple answer...

Documenting and mentally reasoning about exceptions is only half the problem. The other half is scaling this discipline inside individual teams and scaling it out beyond as well.

As a for instance, I can give instruction to my team to document exceptions when they are used, but during code review, I cannot guarantee that my engineers are being thorough enough to also document exceptions thrown deeper in the stack -- especially when dealing with a very large pre-existing codebase they didn't write and only have a passing familiarity with.

documenting every possible exception becomes next to impossible if any function above in the stack decides to throw a new kind after a code change...

Even Java had to create some escape valve for its explicit exceptions, just because there was a huge set of them that could appear literally anywhere.

But then, exchanging that for return values does not make your life easier in any way. It just means you will have to write redundant tests, and will have a flow that is exactly equivalent to the exceptions flow, after you simplify the redundancy down.

You can have your compiler or IDE making the list of possible exceptions for you. But that list is always too big to reason about anyway, and for high order functions, it may be the entire set of exceptions available for your code and some more the it can't discover at compile time.

Haskell has tried a different take on the problem with error monads. It's clean, simple, and does not solve the entire problem. So, it also has exceptions, and they are even worse there than at the imperative languages, because everything is high order.

I think people are entirely justified in ignoring the problem and moving on. Unless you want to dedicate yourself into research, there is not a clear path to solve it.

+1 for this!

Unchecked exceptions subvert the type system, so they are convenient because they are cheating when compared with Result<T,E> types.

Result<T,E> types are the one feature that have me thinking of moving from C++ to Rust. A lot of my experimental C++ code with callbacks looks like a bit like this now:

template <OnSuccess, OnFailure> void someFunction(..., OnSuccess onSuccess, OnFailure onFailure) { ...

Using the above style:

1. Allows error handling to be type-checked 2. Uses the heap less

But it's very clunky! If anybody has a better way to achieve these two things in C++ I'd like to know ...

Edit:

Erlang, which is known for its reliability, gets this right: kill the whole Erlang process in a clean fashion so that we can get transactional semantics. (It's not that performant though.)

> Erlang, which is known for its reliability gets this right: kill the whole Erlang process in a clean fashion so that we can get transactional semantics. (It's not that performant though.)

This is also the rust way - panics kill the thread they occur in.

Thanks - that's good to know! Another reason for me to try rust then ...

Just for fun, really, I've done a Result impl in my lab code[1]. I'm willing to improve it just to see how far I can get with it. :)

The next improvements will be custom error types (I had it done but reverted in this code) and a error_chain like feature (started testing and forgot to remove some not working code there).

[1] https://github.com/fungos/sllab/blob/master/src/viewer.cpp#L...

Thanks for this.

I don't fully understand your code yet: Is it the case that the type-safety of your approach relies on the caller honouring the convention of accessing Result<T,E> via the try and try_chain macros?

I really need to look in detail at how Rust enforces type-safety of Result<T,E>. One thing that I like about how algebraic data types work in functional languages (e.g. Either in Haskell) is that they eliminate entirely the need for null pointers completely and the possibility of derefencing them (Tony Hoare's billion dollar mistake) and I had assumed that Rust would give me the same.

> I really need to look in detail at how Rust enforces type-safety of Result<T,E>.

Basically, you can't use it as a T or an E; you have to process it in some way first. Basically, if you're familiar with Either, you know the whole story, it is the same. We originally had Either, then someone made Result, people always reached for Result over Either, so at some point, we axed Either. There is https://crates.io/crates/either though too.

The type safety of my sample is based on the auto deduction of the types at compile time.

You can see somewhere there I do this:

  return std::move(ArgsResult Ok(options));

I could also have done:

  ArgsResult func() { ... return Ok(options); }

In this example there are two instantiations of the type Result<T, E>:

  typedef Result<ProgramOptions, Error> ArgsResult;
  typedef Result<bool, Error> ProgramResult;

(Note that the typedefs aren't needed, but I prefer to be explicit there.)

The try macro is just trying to emulate try! and it is not really needed. We may access the result by hand and the correct type will be used, but it is way uglier (see: https://github.com/fungos/sllab/blob/master/src/viewer.cpp#L...) due the access on the struct members .e (error) or .v (value). And if I try to assign .v to another type, the compiler will give a compile time error.

And probably would be better to use std::variant there as it is more like a tagged union, I just haven't tried to improve on it anymore.

Also, note that my unwrap doesn't panic, I should put a test/assert there...

Anyway, the poc is there and it should possible to have something almost as clean as Rust with some more work.

Result is exactly Haskell's Either, just with a different name.

> I had assumed that Rust would give me the same

They do. ADTs in Rust are very similar to languages like Haskell. The only major difference I can think of is everything is stored inline in the enums in Rust (they're what one might call "value types" in some languages) without any allocations required.

It's fairly straightforward to have a rust-Result construct in C++. It's a bit more tedious if you want to make it a proper tagged union, but not inherently something that can't be done with a minimum of code and be as safe as possible.

There's this: https://github.com/ptal/expected which seems to work pretty well in my tests so far, though I haven't benchmarked it yet. The default implementation uses exceptions as the error type but you can supply your own type (e.g. an int, string, or a wrapped version of either or both) trivially.

just looked at this, nice! but i wasn't able to find how in the document how to propagate errors up the stack ?

i.e. a parent function needs to expose an error type that is a base class of all errors that can occur the methods being called, right ?

and i wondering how you would get stack traces from these errors ?

Best just to use std::exception as the base class and derive from that, then if you want to get a stack trace just rethrow the contained exception.

I've tried using this and my own implementation of Result<T,E> based on Alexandrescu's original talk, and both are significantly slower that C error codes or exceptions using the test program in the OP.

Thanks for the link. I'll check it out and see whether it works for me.

In case you haven't read it: The Trouble With Checked Exceptions ( http://www.artima.com/intv/handcuffs.html )

Thanks for the link. I hadn't read the interview, though I had read the following by Eric Lippert where he references it:

https://ericlippert.com/2014/03/03/living-with-unchecked-exc...

I'm not really a fan of checked exceptions either for the reasons that Anders Hejlsberg outlines in the interview.

However, the bit where he writes:

"I'm a strong believer that if you don't have anything right to say, or anything that moves the art forward, then you'd better just be completely silent and neutral, as opposed to trying to lay out a framework."

Isn't that exactly what has happened with unchecked exceptions in C++ and C#? They've added complexity compared with C style error codes, without actually moving the art forward.

Destructors make function returns more expensive; exceptions are faster in these examples because they elide returns.

At a guess, destructors would reduce the advantage of exceptions in the case of deep stacks.

I forked this on Github and changed the C code to be more idiomatic, i.e. using error codes instead of allocating a struct (!) and strduping an error string (!!) the way it was originally done. It causes the C version to perform significantly better than the C++ version.

Code is here: https://github.com/julian-goldsmith/excspeed

Do you have numbers? I want to try doing the same test against the proposed expected<T> as well but haven't had the time yet.

EDIT: From your version on a 2013 rMBP, Apple clang 8, I get (top-left 5x5 corner of the matrix)

eccCC ecccc ceCcC eceCc ceeec

compared to the original:

EEEec EEEEE EEEEE EEEEE EEEEE

EDIT2: And switching out 'throw std::runtime_error("Error");' for 'throw int();'

eccCC eEecc eeeec ecCec Eeeee

I ran it on my x86_64 Linux server, using gcc. The full results can be found at https://juliangoldsmith.com/excspeed%20results.txt, but the matrix output looked like this:

  ecCCCCCCCC
  ccCCCCCCCC
  ccccCCCCCC
  cccccccCCC
  ecccccccCC
  cccccccCCC
  ecccccCCcc
  cccccccccC
  ecccccCccc
  cccccccccc

EDIT3: Changing C code to check for error and increment the error variable if it's non-zero, compared to C++ throwing an int():

  ecccc
  eeeec
  eecec
  eeeee
  EEEEE

and against throwing std::runtime_error("Error"):

  ecccC
  eeece
  ceeec
  eeEee
  EEeEe

To be fair, you should also modify the exceptions case to throw a type of exception that doesn't encapsulate a dynamically-allocated string. Otherwise you're now comparing "errors with dynamically allocated string content" vs. "errors that are just a number". Everyone agrees that the latter will be much faster, but there are obviously significant advantages to dynamic content in terms of expressiveness of the errors and ease of debugging. In any case, C++ exceptions are perfectly capable of being just numbers if that's what you want (in fact, you can literally use an exception type of `int`).

I think the point of the article, though, was to compare methods of transmitting the error, not methods of representing the error.

(Also, presumably your change doesn't affect the 0%-error-rate case, where exceptions should still be faster?)

>Also, presumably your change doesn't affect the 0%-error-rate case, where exceptions should still be faster?

In my earlier test, it looked like it was pretty much even between the two. I haven't looked at assembly or anything, but I'd imagine SEH has a bit of overhead.

As for the type for exceptions, I'm seeing what happens when the C++ version throws ints. I'll update here once I have results.

EDIT: Full results can be found at https://juliangoldsmith.com/excspeed%20results%202.txt , graph reproduced below. I've also updated the code on my copy of the Github repo.

  ecCCCCCCCC
  ccccCCCCCC
  cccCCCCCcC
  cccccCCCCC
  ccccccccCc
  ecccccccCc
  eecccccCcC
  cccccccCcc
  eccccccccc
  cccccccccc

EDIT2: I'm also testing it without optimization enabled, to see whether optimization was making the C version faster than the C++ version.

EDIT3: Apparently turning off optimization makes the C version very significantly faster on my platform: https://juliangoldsmith.com/excspeed%20results%203.txt

  cCCCCCCCCC
  cCCCCCCCCC
  cCCCCCCCCC
  cCCCCCCCCC
  cCCCCCCCCC
  cCCCCCCCCC
  cCCCCCCCCC
  cCCCCCCCCC
  cCCCCCCCCC
  cCCCCCCCCC

C++ exceptions, in modern implementations, should have no time overhead when no exception is thrown.

C error codes have the overhead of comparisons to check if the return value is an error return.

EDIT: To expand on this: Modern implementations of C++ exceptions build tables mapping code addresses to information about the exception context at that address. The table is created by the compiler and stored in a separate segment of the executable. If no exception is thrown, the table is never consulted. When an exception is thrown, the exception-handling code consults the tables while unwinding the stack. For each return address, the EH tables tell it what destructors need to be run to unwind that stack frame. So, there's no need for comparisons or branching in the code itself. Of course, table-driven execution is expected to be slow (for the same reason interpreted languages are slow), but the idea is to optimize for the common case where there are no exceptions.

I looked closer at the C code and noticed that it's not really realistic error checking (both before and after your changes). Each level of function is simply tail-calling to the next level. No level is checking whether the return code indicates an error. Instead, an error pointer is passed all the way down the stack and then possibly filled in or not at the bottom, but never inspected in the intermediate frames. This means all the extra branching that is normally associated with error code checking is being skipped in this test.

It's not surprising, then, that you're seeing the exception-based code and the error-code-based code performing the same (one or the other winning at random) in the 0% error rate case -- they are actually the same code.

In a more realistic test, each intermediate function would call the next function, then inspect its result to see if it indicated an error. If so, the intermediate function would return early (probably passing along the same error code). Otherwise it should do some additional processing on the result (maybe increment it or something, for testing purposes) before returning. In the C++ exceptions version, the error code check could be skipped.

With all that said, I would still expect that when exceptions are actually thrown, they should be slower than error code checking.

This is a useful topic for empirical study, because there are a lot of factors that can change the outcome.

It's not just Java that has big stack traces; it's more complex programs that combine multiple different abstraction layers that have big stack traces. The easier it is to use third-party code, the more likely it is you'll have big stack traces, because it encourages more modularity, and modules tend to be better factored and less flattened with respect to the original programmer's intention and code that gets things done.

Another variable not talked about here is hot code paths. Exceptions can be slow when first used, because they can touch code or data that has been (optimistically) removed from the main code path. This is usually done under the rationale that exceptions are very rare, and that making code have zero-cost exceptions is very important. E.g. exceptions on Windows x64 use table information, while exceptions on Windows x86 use a linked list of stack-allocated structures with callbacks, pushed and popped by exception handlers. So you'd expect the first exception of many thrown to have a higher cost in the x64 model, while the throughput without exceptions to be lower in the x86 model.

Ironically, manual error propagation up the stack can't be so easily taken out of line, so it affects throughput more.

Another thing that is missing is cleanup. When you have objects that need to be destructed at different levels of the stack, the destructors need to be called during the stack unwinding that happens when an exception is thrown, and I'd expect it to fill up the gap between exception throwing and error propagation.

I wonder how often the argument "exceptions are slower than error codes" actually means "exceptions are slower than having no error handling at all"...

My semi-educated guess: Far more often than any of us is comfortable admitting...

The title is misleading. Their "error codes" are not simple C error codes but are rather structs which they are dynamically allocating upon error occurence. Their "create_error" function below is very inefficient compared to using traditional int error codes with a lookup table for the messages.

   struct Error* create_error(const char *msg) {
    struct Error *e = malloc(sizeof(struct Error));
    e->msg = strdup(msg);
    return e;
   }

Nice to know but arguably real-life C++ code using error codes written today won't (shouln't) look anything like their C code. Ok it might not change the outcome of the performance tests but still it would imo be more interesting to see the results of something like

  llvm::ErrorOr< int > func0() {
    const int num = random() % 5;
    if(num == 0) {
      return func1();
    }
    //etc
  }

Also I'm not too sure the 'because call stacks are usually not this deep' (for depth of about 66) claim is justified, depending on the type of application.

That jumped at me too. A pointer to a pointer to a struct is not a very common error code strategy.

interestingly this is similar to how Rust does it: it has a Result<Type, Error> trait.

https://doc.rust-lang.org/std/result/

I think Rust got it from OCaml:

https://realworldocaml.org/v1/en/html/error-handling.html

With the try! macro, it's got the test and return built in. Further, if the return value is large, the caller will allocate space for it and pass the address of the space as a hidden parameter. If you've got rich return values, the final code will be pretty much isomorphic with the example C code (it wouldn't be a pointer to a pointer, though; it would be a pointer to a struct).

Note that Rust now also has the ? operator, which can be a bit easier on the eyes than try!:

https://github.com/rust-lang/rfcs/blob/master/text/0243-trai...

https://blog.rust-lang.org/2016/11/10/Rust-1.13.html

FWIW, C++ has this really cool feature they call "exceptions", which is a language-level implementation of Result, try!, and ? that is automatically included as part of any function that would otherwise have to manually unwrap and pass along an error in languages like Rust; a really great bonus is that because the compiler has implemented this feature and abstracted it from the developer, in addition to it removing a lot of boilerplate from your code, it allows for an extremely fast table-based implementation they call "zero-cost exceptions" that can almost always completely eliminate the CPU cost of having to constantly check and forward error information.

> really cool feature

Arguable.

Yes, people are quite aware of that, and if you check the comments you'll see points made against it. Also, ? is very much language level.

Condescending sarcasm is unwarranted. People are well aware of what exceptions are, and the arguments against them amount to more than just performance.

I've implemented a Result-like error code construct in plenty of C++ projects and disabled exceptions precisely because I don't think they're a good way of handling error states in code.

I'd expect the compiler to allocate space for ErrorOr on the stack of the caller and pass an address to this space in a hidden parameter. In effect, the implementation func0 would probably be typed a bit like this:

    void func0(llvm::ErrorOr<int> *__result)

An advantage of passing a pointer to the error that you may be missing is that if you can pass this location all the way down, you can eliminate redundant copies / moves / assignments. This is harder when the return value and error value are combined in a single tagged union, so there will be a lot more data shuffling code all the way back up the stack.

  struct Error **error

This is the most asinine error handling strategy I ever met. I'd choose C++ exceptions over it any day, no matter if exceptions are slower.

On the other end of spectrum is using good old error codes, not reinventing exceptions on your own.

If you allocate data for your errors dynamically (perhaps they have formatted strings for informative messages), you'll want to return a pointer, otherwise you'll have copies throughout; and if you get passed a pointer to a pointer, you can save lots of shuffling, as I pointed out elsewhere on this thread.

There is a solid rationale behind it.

You don't save any shuffling, a pointer to a pointer is still a pointer.

If you allocate data for your errors dynamically (perhaps they have formatted strings for informative messages), you'll want to return a pointer, otherwise you'll have copies throughout

Not sure what you mean exactly but I don't think you need copies all over the place since this is C++ so instances can be moved and there's also copy elision. And should you use 'int fun(Error& error)' with error containing an std::string for the message then at least the principle of 'you need a pointer' is dealt with by std::string insetad of having to do it manually.

There is a solid rationale behind it.

Might be but from a usage point of view it seems harder to deal with than alternatives (see 'ErrorOr' in another comment, or passing 'Error& error' or so: these days raw pointers are frowned upon, imo with reason, and raw pointer to pointer doesn't make that any better). Suppose I write an

  int fun(struct Error **error) {
    if(somethingWentWrong) {
      StoreErrorCode(error);
    }
  }

In StoreErrorCode I'd first have to check if '* error == nullptr' or not. If not I'd have to '* error = new Error()', I guess. If '* error' does point to an instance, what do I do? 'delete(* error)' followed by new? Or reuse the current instance? That's imo too many questions already and I didn't even store an error code yet. Sure the answers to those questions can be dealt with in an error handling API used by the library. Or one could rely on some convention that '* error' always is NULL and hope everyone follows that convention. But unless somehowe proven that this is really really needed for performance reasons I'd pick any easier to work with (and hence less likely to introduce bugs) C++-style alternative over pointer to pointer anytime.

This is common in Objective-C APIs but Objective-C will “auto-release” the error objects and attempting to send a message to a "nil" object does nothing at all. Even better, Swift bindings will recognize that particular idiom as one of the supported ways to handle errors for use in Swift statements.

Therefore I would say this approach is bad for C and C++ but not bad in general.

The obvious performance advantages of zero-cost exceptions combined with the equally obvious benefits of "the vast majority of code, hopefully even almost all code, doesn't need to have error checks all over the place" is why I am really annoyed at the Rust crowd for forsaking them, which to me sounds mostly like "I am scared of language features in a similar manner to the stereotype of a Java developer" (which is almost ironic given that when you write Rust you are supposed to make your code exception safe anyway, which kind of defeats the only benefit to having avoided them).

Error checks are almost free and can be considered zero cost because branch predictor works really well with error checks (because errors are rare). Branch predictor uses some space in L1 cache to store tables so it's not 100% free but very close to that. Exceptions aren't zero cost either but you're paying for them only when an error occurs most of the time.

Rust didn't completely forsake them--they were just renamed "panics". The difference is that the language and ecosystem push you to only use panics for truly exceptional cases, and to use Result or Option for expected failures.

Exceptions are for exceptional events. If exceptions are effecting the performance of your application, you have done something horribly wrong.

My sentiment is a relative of yours: I care more about the performance of the two methods when there is no error, as that is the common path. Hopefully. In my own brief micro benchmarks, exceptions do have a performance penalty when there is no error. With that said, I still use them, but I sometimes have to think carefully about when.

If you're seeing performance degradation as a result of including exceptions in your application the exceptions are generally not themselves to blame.

The surrounding code can be a culprit. For instance, if you have a conditional that's not so easily predicted that in turn throws an exception then the cost comes not from the exception but from the branch misprediction.

It all boils down to orders of magnitude. The overhead of a branch + exception is going to be a few cycles. That said, only in a really hot code path (eg; vectorized matrix operations) where I'm really pushing FLOPS then I'll avoid throwing exceptions. Realtime code is another place where avoiding exceptions is wise (since their cost is usually negligible but non-constant due to the exception frame). Otherwise, I have yet to see a benchmark that shows detrimental performance loss in all but the hottest loops.

I work on (among other things) a soft-realtime streaming runtime where we care about microsecond latency, so I care about the cost of the common data path. You made a good point about the surrounding code being more simplistic making it more amenable to optimizations. But, I still see that as the "cost" of exceptions.

Yeah, I also work on systems where we care about microsecond latency (market data handling). In most cases, we still leave the exception handling in. It's maybe a few cycles of cost and we definitely need the whole system to stop if any of those exceptions come up. Then again, the costs are minimal b/c our conditional to check for an exception is easy to predict (simple comparison to a stack variable probably on the register already).

In a hard realtime system you really have to move all the exception handling upstream and use error codes downstream. Problem is this does hurt performance in the average case.

This is partly why I really want something like the expected monad to be accepted into the C++ standard: https://github.com/ptal/expected

I wonder what the performance difference is when you're actually handling the errors though. I'd imagine that

    try {
       doThing();
    } catch (Exception1 e) {
        handleError1();
    } catch (Exception2 e) {
        handleError2();
    } catch (Exception3 e) {
        handleError3();
    }

Might be slower than

    Error err = doThing();
    if (err.code === ERROR1) {
        handleError1();
    } else if (err.code === ERROR2) {
        handleError2();
    } else if (err.code === ERROR3) {
        handleError3();
    }

Exceptions are best in my mind when they are very coarsely-grained, and error-codes best when you need to handle very fine-grained errors. This is certainly true in the code ergonomics, but I'd be interested in seeing if it's true for performance as well.

Ok, so sometimes one is slower, sometimes faster. One interesting variable to look at is how much faster/slower exceptions are. Are we talking orders of magnitude ?

Last time I used exceptions was in the year 2000 (I kid you not). I remember that after removing the exceptions, the socket code was several times faster.

I still don't use them mainly because of the unpredictability of the control flow.

These days, exceptions are usually implemented so that code with exception-handling code is no slower than code without it, in the case where no exceptions are thrown, since that's the common case. These are called "zero cost exceptions."

It's a rather misleading term, because the "zero cost" only applies when there are no exceptions thrown! You only get "zero cost" when you go through a try block without throwing.

The tradeoff is that throwing an exception is pretty slow. Since all of the normal code is built as if no exceptions happen, throwing an exception has to go through and carefully unwind all the stuff currently in flight before it can start executing the catch.

If your code very rarely throws, you'll be faster using exceptions. If it throws frequently, then you could easily end up losing an order of magnitude or three in performance.

It would be more interesting to have this measurements take place across a wider selection of C++ compilers[1}, not just the two usual open source ones.

[1} - https://en.wikipedia.org/wiki/List_of_compilers#C.2B.2B_comp...

If you have access to more compilers, the test code is here: https://github.com/jpakkane/excspeed

Doesn't seem something that would work out of the box on Windows, I will need to look at it in a few weeks time for the VSC++ compilers I have installed.

The problem is throwing. Once you start throwing on your critical path, which will happen as the codebase and number of developers expands, you're in for a rude awakening.

Reply on news.ycombinator.com