Hacker News Reader

Comment by nicolaslem | Hacker News Reader

I often find myself using "except Exception", while I don't like it, I cannot find another way.

For instance when making an HTTP request. The requests library can throw thousands of different errors (SSLError, BrokenPipe, socket errors, errors from urllib, errors from requests itself...).

As far as I know, it is the only choice if you want to know if a request succeeded or not so you can deal with it without your view returning a 500.

I would really appreciate if someone has another solution to this.

View on HN

IMO, there's nothing wrong with "except Exception" if your handler is, in fact, meant to handle all exceptions.

(Well, there's one thing wrong, which is that certain standard library errors in Python 2.x don't derive from Exception [socket error, I'm looking at you]. This has been addressed in more recent pythons, but there's still lots of 2.x out there.)

Python's rules for handling exceptions in a class hierarchy are designed to provide programmers with the ability to handle errors at whatever level of granularity makes sense. If your requirements say that the software should treat SSLError and BrokenPipe differently, then by all means, handle them separately.

More often, however, you're making a library call for which any failure has identical requirements. When you're trying to read a configuration file, you probably don't care whether a hard drive is corrupted, has a loose power cable, is on fire, or somehow triggered a divide by zero error somewhere in the bowels of an I/O library.

  >>> import socket
  >>> issubclass(socket.error, Exception)
  True

It's been like this since Python 2.4 at least.

I stand corrected. Well, partly. Looks like it was changed in 2.6.

https://docs.python.org/2/library/socket.html#socket.error

All of requests' exceptions inherit from requests.exceptions.RequestException [0], so you could catch it.

[0]: http://docs.python-requests.org/en/master/_modules/requests/...

Kennethreitz requests lib? Just use requests.exceptions.RequestException, that should handle every error, if not then report it.

For python urllib IOError and OSError should suffice. At least in py3.

and in Python >=3.3, IOError is an alias for OSError.

  import urllib.request
  try:
      urllib.request.urlopen('https://wrong.host.badssl.com/')
  except (IOError, OSError):
      pass

causes:

  ssl.CertificateError: hostname 'wrong.host.badssl.com' doesn't match either of '*.badssl.com', 'badssl.com'

You should have (+/-) one "except Exception" in your code which would be the "catch all/log error" thing.

But for how to do with errors, see the sibling comments to this one, and good libraries will give you an inheritance tree of exceptions

This is basically the equivalent of returning a 500 or failing a task. You know that your code failed somewhere but there is no way you can recover from it (granted your code is not idempotent so retrying everything is not an option).

By catching all exceptions as close as possible of the request you make, you can at least do something about it.

As for the tree of exceptions, it works well for a while until networks become unreliable, until remote peers start presenting certificates broken in subtle ways... You end up adding ranges of exception in the except clause as new ones are popping on Sentry. Then you get tired of it so you catch "Exception".

This is the unfortunate story of every single project I do that deals with network calls on the Internet.

Where 'in your code' is a module. Then log the exception and raise a new exception with a module-specific code. requests.exceptions.RequestException is a good example.

My modules do catch SystemExit and KeyboardInterrupt before the bare exception and simply re-raise those.

Good unittesting can trigger the relevant exceptions. I know this can be a lot of work to write tests which trigger all your corner cases. I practiced this in a recent project but now I'm much more confident in my code.

Another strategy is to catch the "expected" exceptions first followed by a general exception handler. This can use the `traceback` module to log the call chain and error message.

A better way is to implement a logging function, which will be recording unknown exceptions the way you prefer: sending an email, writing to a text file, whatever. Then you just write:

    try:
        [...]
    except Exception as e:
        log(e)

It's not much longer than `except Exception: pass` and you gain more control over your system.

Requests cannot throw errors from urllib because Requests does not ever invoke urllib code.

Any error that does not inherit from our top-level exception is a bug: please let us know so we can fix it.

I am working on a crawler that uses async/await, and from experimentation the list is:

except (aiohttp.ClientError, aiohttp.DisconnectedError, aiohttp.HttpProcessingError, aiodns.error.DNSError, asyncio.TimeoutError, RuntimeError) as e:

I want to continue running no matter what, so I also have an "except Exception", but I have better logging now that I know what the known vs. unknown exceptions are.

My pattern on this stuff is to initially write:

  try:
      do_something()
  except ExpectedException:
      handle_it()
  except OtherKnownException:
      handle_that_too()
  except Exception as exc:
      log.exception('New kind of error just happened')
      handle_fallback()

Yes, that's basically what I'm doing.

>I often find myself using "except Exception", while I don't like it, I cannot find another way.

This (pokemon exception handling) is a massively, massively destructive anti-pattern.

Once a code base grows beyond a certain size it usually leads to failures which manifest themselves with strange behavior that takes hours (sometimes days) to track down the real source of.

    try:
        ...
    except (NameError, AttributeError, TypeError):
        # likely programming error
        raise
    except Exception:
        ...

Not pretty, but maybe better than catching every exception.

Reply on news.ycombinator.com