Exhaustiveness Checking with Mypy | Hacker News Reader

Exhaustiveness Checking with Mypy

hakibenita.com

85 pointsrbanffy19 comments

I also find type narrowing extremely useful for error handling: you can use Union[Exception, T] as the result type, and ‘pattern match’ with isinstance, which works in runtime and statically checkable by mypy.

In addition, covariant Union type allows using it with generators (i.e. yield Exception("some error"), and consuming without extra boilerplate. I write more about it here: https://beepb00p.xyz/mypy-error-handling.html#coderef-simple...

This may seem like a good idea, but it gets problematic since you need to always use isinstance checks. Its actually discouraged to return union types by the mypy team: https://github.com/python/mypy/issues/1693

Yeah, but you need to test whether the result is an error or not in some way. In languages with better ADT/pattern matching support (like Haskell or Rust) you'd use case/match operator, or whatever it's called.

I've read the issue, and it seems that it's more about the lack of overloading/dependent types, e.g. their example with pow function. In c++ you'd have two overloads:

    float pow(float, float)
    int pow(int, int)

Whereas in python, even though you can achieve the equivalent behaviour in runtime, mypy (at least until recently) couldn't express this, so you had to define something like

    def pow(base: Union[int, float], power: Union[int, float]) -> Union[int, float]

Which might be annoying on call sites when you do expect the return type to be int (but there is no way for mypy to know that with such signature).

In my case I do want to return a variant type unconditionally, so this warning doesn't really apply. It's more similar to an Optional type (just instead of None I am using a more meaningful Exception): while it's not great if your code is riddled with optional return types, sometimes you don't have other choice but using it.

You can use typing.TypeVar for the `pow()` example. TypeVar was added via the original type hints PEP in 2014.

    from typing import TypeVar

    T = TypeVar('T', int, float)
    def pow(base: T, power: T) -> T: ...

Python's static type system can express what you describe using TypeVars or overloads. The issue is that `int pow(int, int)` is fundamentally incorrect. If the exponent is a negative integer, the return type would need to be float.

As evidenced by your comment, most people forget about negative exponents. The vast majority of usage is positive exponents; people expect an int back when they do `pow(2, 2)` and returning a Union or float would result in annoying false positives.

Issues like this are another reason Python should have a native switch...case control flow instead of trying to cram it into a chain of elifs or a dict. In other languages, calling a switch on an enum will cause the type checker to ensure that all cases are handled or that there's a default case. I see there's a PEP out to push this functionality into elifs, but it still seems inelegant to me.

The `match` statement, slated for the next version of Python (3.10), is rather comprehensive. Don't think we could reasonably expect much more than this to be retrofitted into an old language [0]

https://www.python.org/dev/peps/pep-0622/#exhaustiveness-che...

https://www.python.org/dev/peps/pep-0634/

[0] Trivia: Python is older than Linux!

I'm so excited for match to become a thing. Honestly, exhaustiveness had kept me from using Python for some work that I'd have otherwise loved to use it! I enjoy FP and some of the other benefits of Ocaml, which is what I use for programs where Mypy can't give me what I need. But at the end of the day, I'm just not as nimble with it, and I can just see the python in my head so much easier.

Very exciting for experienced developers missing this from other languages, but I predict the growing spec of Python is leading it away from the "my first language" crowd in future.

Yet python never needed a case statement. It used to be simple to write and read language, unlike so many others.

Cramming in new statements just to check Enums types properly sounds... well wrong... The types don't do anything except make code harder to read. But I know that unpopular opinion here... I guess I just like old simple Python more.

Curious what other languages you're fluent with? I found once I'd grokked a language with `match` (Rust), I missed it when writing Python afterwards.

In my professional career I've worked mostly with C++, PHP and Python and these are probably ones I'm most fluent with. But also done tone of Perl, Delphi, VB and C# back in the day. Strange mix.

However I also think that how you look at the language greatly depends on what actual projects you have been working on and with whom plus personal taste. I've been mostly in finance and accounting and personally love code that is very easy to "read" (definition of readable code seems to vary per person).

Example: C++ has so many things crammed into it that I don't see the value of it for doing stuff that are not high frequency trading. It is like using a sledgehammer to drive a nail in the wood.

Looks like you haven't had much immersion in functional languages - the ones you list are all quite an imperative flavor. If you're interested in sampling some other styles of language I suggest the book "Seven languages in seven weeks"

... I never done LISP in professional life but I have learned it and used it for side projects. I've read SCIP and found that everything said there can be applied to most programming languages. I don't understand what you're comment is supposed to imply? That I don't understand match cos I didn't write first reddit version?

I've spent 25 years discovering languages and concepts in computer science. I've even played with Smalltalk, Ada and some Algol variant. Even wrote COBOL for short while. What's your point?

On one hand, this "feels" useful ... on the other hand, I can't shake that unnerving feeling that this kind of exhaustiveness check should belong in tests instead.

If the issue is to help you ensure that you check your edge cases properly, and ensure you are acting within specification, then presumably you should be addressing these issues in your tests, not in production code.

Therefore the whole exercise of writing code to catch errors of this kind 'at compile-time vs runtime' is somewhat of a red herring, if not dangerous.

Having said that, I'm also guilty of frequently writing

    else: error("In theory this should never happen")

Heh, I feel you. Test were indeed used to verify that your code runs correctly and with edge cases, and stuff like isinstance were discouraged so you wouldn't need to think about types but rather of interfaces. However I assume that a large influx of people who had experience with typed languages had changed general idea of how Python should look and be used. Oh well, times change and we all allowed it so... types it is :/

> presumably you should be addressing these issues in your tests, not in production code

What you may be missing is that this code will be guaranteed by MyPy to never actually run in production. In fact what it's effectively saying is "if this code can run, there's a type error", and therefore MyPy forces you to ensure that the code can never run.

This is great! This function belongs in the standard library.

See relevant section in the article:

https://hakibenita.com/python-mypy-exhaustive-checking#the-f...

Reply on news.ycombinator.com