Using Mypy in Production

This would be flagged as unreachable I believe.

mypy/pyright also supports exhaustive checking with unions

  def get_activation(key: Literal['sigmoid', 'tanh']) -> nn.Module:
    match key:
      case 'sigmoid': return nn.Sigmoid()
      case 'tanh': return nn.tanh()

[1]: https://en.wikipedia.org/wiki/Disjunctive_normal_form

    @dataclass
    class Message:
        event: Literal['message']
        msg: str
    
    @dataclass
    class File:
        event: Literal['file']
        url: str
    
    typedload.load(data, File | Message)

After the load, you can trust that your objects are well formed and let mypy do its thing.

So types can be useful at runtime as well. Otherwise the alternative would be to use raw dictionary, which mypy can't check.

pydantic does a similar thing but it uses its own different typing, so it needs a mypy plugin or it flags everything as wrong.

Code like the first version is usually someone is trying to make the API more convenient, but perhaps a better way, that's more in line with static type checking, would be to have a separate `activation_from_str(...)` function.

For a similar situation, I was thinking about doing something along these lines (somewhat Rust inspired), although the duplication of variant names in the from_str method isn't ideal:

    import enum
    import typing

    # stubs so this example is self-contained & runnable
    class Sigmoid: pass
    class Tanh: pass

    class ActivationType(enum.Enum):
        sigmoid = Sigmoid
        tanh = Tanh

    class Activation:

        @classmethod
        def new(cls, variant: ActivationType):
            return variant.value()

        @classmethod
        def from_str(cls, variant: typing.Literal["sigmoid", "tanh"]):
            return cls.new(ActivationType[variant])

    activation1 = Activation.new(ActivationType.sigmoid)
    activation2 = Activation.from_str("tanh")

    >>> l = Literal["foo", "bar", "baz"]
    >>> "foo" in l
    TypeError: typing.Literal['foo', 'bar', 'baz'] is not a generic class
    >>>    
    >>> isinstance("foo", l)
    TypeError: Subscripted generics cannot be used with class and instance checks

That's just how effective Python is done.

All that said, Mypy is far worse than TypeScript. I think Mypy would be so much better if it improved support around dict inference. It shouldn’t just say “I dunno, it’s a dict of strings?” but rather infer an anonymous TypedDict

The problem with Python is that most code is shit. Sorry, but "fighting the compiler" because you want to return different types based on some flag isn't the compiler being annoying, it's the compiler exposing bad code.

It's not until before summer typing for django actually started to work somewhat. All stubs did was to make mypy not complain, but not until now it actually can catch some errors.

Getting a python project typed is a world of pain, and in the end you can't really trust it anyways because of all the hacks making the type-checker happy.

1. Most of the Python community doesn't really get the importance of static typing. As a result tons of dependencies completely lack typing, or are written in a dynamic way that is incompatible with static typing.

2. You have to go to some significant effort to set up static type checking in CI, and because of the above issue getting 0 errors is much harder than in e.g. Typescript.

3. The semantics of type hints are not defined so there are multiple incompatible interpretations. Better hope your dependencies use the same type checker as you!

4. MyPy is way better than nothing but it's actually pretty buggy and absolutely full of unsoundness and legacy hacks (not that surprising given its age).

I would strongly recommend using Pyright if you make the unfortunate mistake of starting a new project in Python. It is much better than MyPy and the author is some kind of bug fixing robot.

Cognitive Load.

I guess implementing multiple protocols (interfaces in other languages like C#) is possible and more awkward using ABCs?

The runtime validation stuff has been interesting to watch. Do you use Pydantic? It's very popular but I have a hard time getting over its willingness to cast / coerce implicitly (if I mark a field as an int, and pass in a str, I want an error -- is that weird of me?).

Yeah, this drives me nuts as well. But Pycharm has recently started inferring types and jump-to-declaration works, so I believe obtaining that information must be possible.

Celery is another library in the group like django/pytest in that everything about it is suuuper dynamic and *kwargs-y. It drives me up a wall that I can't readily decouple task functions from their interfaces in a typesafe way. Also I don't know how to decouple the tasks from a Celery app with a pre-defined broker uri without resorting to config files/env vars - that approach simply does not unit test well.

While not without caveats, but I started experimenting this, and it is quite useful.

> The main thing that eventually forced decision was a flaky (depends on cache) mypy crash using paramspecs half a year ago. At time paramspec support was still in progress and there’s a good chance that specific issue is fixed.

I actually think I ran into this exact issue (ParamSpec-related, only fails when reading from cache, etc.), which led me to pin a Mypy development version for a while and was fixed recently.

But other than that, I generally prefer pyright day-to-day because the experience using VSCode is far better.

I think "readability" is subjective so I just thought I'd throw my own opinion in there.

Libraries, ecosystems and actual business reasons that matter aside, if I cannot have that, I would rather use something like Kotlin instead.

PHP does it better. Types are an add-on, but integrated into the interpreter. Also it has never been pretty in the first place, so nothing was lost.

This makes me think you've never dealt with untyped Python code in a large production repo.

The frustration seeps in the hundredth time you encounter an untyped `get_user_ids(...)` written by a co-worker (or yourself in the past), and wonder "Hmm, is this going to return a list of ints? Strings? UUIDs? A generator of those things?" and then you have to dig into the function to find out what it's actually going to give you, and then each function inside of that will have the same issues, and pretty soon you're building a mental model of a call stack to discern types anyway, you'll be happy to embrace the type system of python.

"Readability" doesn't matter if you're unable to simultaneously quickly build a mental model of what is actually happening in the code. Sure, `def get_user_ids(...)` is super readable in that I quickly understand "this is going to give me some user ids", but that's useless when you consider the immediate next step of "what format are they in, so I know what I can do with them afterwards". `def get_user_ids(...) -> Optional[Iterable[int]]` is infinitely more useful, because I know that I can't use the function unchecked inside a list comprehension, because it might return `None`. And if it doesn't return `None`, I know I have, for example, integer user ids which I can compare with `==`, vs a custom type which might not implement `__eq__`.

For instance, if I see a function signature like "def parse_int(x: str) -> int", I can probably guess what the function will do. It might be parsing the entire string as an integer, or it might be parsing the first integer it can find within that string, but that type signature by itself eliminates a lot of weird things that could plausibly fall within the purview of a function named "parse_int". It can't return an iterable over all integers parsed from the string; it can't conditionally return an int if there's a single integer and a list of ints if there's more than one; it only accepts a decoded string (i.e. not bytes or any other string representation); it can't take a list of strings and return the first integer it finds; it can't return a string representation of the parsed integer; it can't return None if parsing fails (meaning it will likely throw an exception instead); and so on. Thus, even if I need to read the function body to see exactly what it's doing, I go into it thinking "in what specific way is this function parsing a single integer from a single string?" rather than "I wonder what this function does".

If you know how many parameters it's easier.

If parameters have names that you can understand it's even easier.

If parameters even let you know what kind of value they want it's also easier.

When I read python I'm not reading a novel. I want to know how to use a function, and possibly I want to know that without reading the function.

So it might make 1 line harder, but makes me skip reading 50 lines.

This hits the nail on the head for me with Python. The standard libraries take a very “batteries included” approach but the language constructs don’t, and that means people use the flexibility of the language to do things their own way. I find that really increases my mental workload when reading other people’s software (how have they done this) and when writing my own (which way should I do this).

I tried to acknowledge the deficiencies in LOC in the second paragraph... but it felt like a useful shorthand for conveying a sense of scale and complexity (at least to an order-of-magnitude) in a domain-agnostic way.

Outside of Mypy, we use: black, isort, flake8, docformatter, and autoflake (to remove unused imports) -- all on pre-commit + enforced on CI. I'd like to see Black cannibalize more of that toolchain :)

> Sometimes conciseness is better than cleverness.

Isn't concise and clever on the same side of the balance? Cleverness and abstraction is how you cut down on LOC.

Abstraction has its own cost, so while you can trade LOC for cleverness/abstraction, there's limits/costs to that as well.

Also also, python/mypy still lacks support for higher-kinded types, which really puts a soft-ceiling on the amount of abstraction you can type-safely utilize.

Aaaanyways, I think the whole original point is, "we have a large codebase, and mypy helps keep the wheels on at speed".

+1 on black and precommit. That has done wonders for codebase consistency. I'd run flake8 too but the maintainer is weirdly opposed to supporting pyproject.toml for config.

a few more things like that and it will start to look like machine generated javascript

And there is nothing wrong with using a type checker for your Python type annotations.