A Failed Experiment with Python Type Annotations
mortoray.com
mortoray.com
No, the workaround is to stringify "Node".
class Node(Object):
def add_sub(self, sub: 'Node'):
...
def get_subs(self) -> Sequence['Node']: #Or maybe 'Sequence[Node]'
> This complexity helped drive the introduction of the auto keyword to C++. There are many situations where writing the type information isn’t workable. This is especially true when dealing with parametric container classes,
This is absolutely wrong! You cannot annotate a function as returning `auto` in C++. `auto` and its variants in other languages are useful for eliding redundant type declarations. Especially long ones that use generics/templates. `auto foo = MyContainer<Tuple<int, str>>();` or whatever is nice than having to double write the type declaration. But, in C++ or java, if you write a function that returns a Mycontainer<Tuple<int, str>>, you have to write that in the function.
This is an intentional choice: function declarations are your apis, and explicit and clear APIs are useful for human readers of your code. That's why you should annotate the public apis[0] even when you can have them be type inferred. Speaking of which, if you want type inference, check out pytype[1], it's like MyPy, but does do type inference on unannotated code. But you should still annotate your public apis. It serves as a sanity check that you aren't accidentally returning something you don't expect.
And of course, being familiar with the differences between iterables, iterators, sequences, containers, etc. is not a bad idea.
[0]: https://google.github.io/styleguide/pyguide.html#3191-genera...
Maybe I'm misunderstanding something, but one can write a function auto foo() in C++ and the return type is inferred. Lambda functions infer the return type by default.
This excludes, as you mentioned, the return types of a function, its parameter's types, as well as the types of fields in a class.
1. Is gradually typed,
2. Doesn't infer types, and
3. Is strict, in that it doesn't allow operations that change types.
It was originally developed as a replacement for mypy that was faster and scaled better to very large codebases.
You can do that since C++14.
The good thing about inference (especially with a REPL) is you can write it without the annotation, and then use the inferred type (in Haskell, I usually find that when I resist the temptation to explicitly annotate types, the actual types are more general than I would have specified.)
But my coding in Haskell is pretty much personal and toy projects; I like the approach, but it may not be ideal for coding in anger.
Inferred, loose parameter types are great, because while I think it's good to know the differences between iterator, generator, iterable, sequence, container, and so on, it's difficult, and inference will give you the loosest one. But with return values, the inferred type will be the tighest, in other words your function will return a Dict (or worse, a defaultdict) instead of a Mapping or MutableMapping. You almost always want a Mapping.
Some things I found useful:
1. from __future__ import annotations
As I mentioned in another comment, this lets you write annotations naturally without worrying about when something gets defined.
2. typing.TYPE_CHECKING is only True while type checking.
This allows you to conditionally import files that would otherwise cause a circular dependency so that you can use the symbols for annotations.
3. def foo(bar: str = None) -> str
If an argument defaults to None, then mypy will recognize that its type is actually Optional[whatever]. So in the example above, bar is an Optional[str]. (I'm not sure if this is mypy-specific.) Optional[T] is equivalent to Union[T, None].
4. You can make typedefs easily.
MyType = Union[Sequence[str], Dict[str, int]]
Makes writing complicated annotations easier.
Annotations in comments or in type stubs are supported in python2 (you can look at typeshed for typing.Text and conditional py2/3 stuff). There's also some other cases, but they're...unique.
The docs talk about this here ("class name forward references"): https://mypy.readthedocs.io/en/stable/kinds_of_types.html#cl...
from __future__ import annotations
The language could just as well recognize Node as it recognizes 'Node', but instead it given as a burden for the human programmer to handle.
Is it maybe because mypy is not really part of the Python language, and can't really make changes to it?
def foo() -> Bar:
...
class Bar:
...
I'm saying the language could work differently.
The case of referring to a class inside its definition is quite different from your "Bar" case. The thing referred to is already declared, it's just not fully defined yet.
There is no logical reason I can see making it impossible for Python to honor the obvious intention of the programmer here. It seems it has just chosen not to do so. Though I'll admit I haven't thought through every weird corner case :)
As rcfox said, you can already enable this behavior, and it will be included in Python 4.0.
mypy isn't behind (here), it's ahead.
I can't find a more authoritative source right now, but I also remember seeing the same thing that this SO answer says - that it was a deliberate choice not to do this so that people could incrementally add type annotations: https://stackoverflow.com/a/38775381
There are also some separate tools that can infer and generate the annotations for you. I haven't tried them personally, but the mypy docs recommends MonkeyType: https://mypy.readthedocs.io/en/stable/existing_code.html#aut...
I recommend checking F# if you want mostly inferred, strong static types and terse pythonesque Syntax (few braces, significant whitespace). Also much faster.
Considering you can use all of .NET from F# you can benefit from the work on C# as well.
What's more, while I found mypy useful myself in several instances, it's still cumbersome to use:
- you must know to use the magic command line arguments to avoid mypy complaining about other libs and imports all the time
- you need to use List[], Dict[], Set[], Iteratble[], Tuple[], etc. instead of list[], dict[], set[], iter[], tuple[], etc., which means an import in almost every file and a very unnatural workflow.
- you need to use Union and instead of |. This is ridiculous. I have to do:
from typing import Union, List
def mask(...) -> List[Union[bool, int]]
def mask(...) -> list[bool|int]
Now, mypy has improved a lot. It's way faster, shows many more things that before, way less false positive, and has support for duck typing and dunder methods (named "protocol").
But it's sad to thing TypeScript is actually easier to use. Having a JS tech easier to use than a Python tech is a good sign we can improve things.
Type annotations/mypy, especially when coupled with dataclasses/pydantic has been very helpful in maintaining a rather substantial 3.7 codebase.
What size codebase is substantial?
The fact remains though that Python is an excellent general purpose language, easy to learn, almost universally known, has libraries for everything (and that’s just the stdlib). It is well suited to rapid prototyping, scripting, and data wrangling. Adding incremental typing makes it even better.
* worse performance
* lack of immutable vectors, alleviated via style conventions
* rarely, awkward lambdas
vs.
* wide pool of people familiar with the language
* lack of JVM lockin: Scala native is not there yet, the library ecosystem is all JVM.
* [good] batteries included
* much better reflection
* access to modern ML
* gradual typing
* no actors
Rust: Manual memory management is verbose and distracting.
Go: Manual exception handling is verbose and distracting.
Kotlin: JVM lock-in.
Swift: Apple lock-in.
Lambdas in Python are quite limited, but offset by natural nested local functions.
Regarding performance, that's not always an issue, and depending on the use case can be addressed with async/await or using a multiprocessing pool. I have had issues with multiprocessing and some DB libs ib the past, but recently have had pretty good success.
Multithreading is still an issue because of the GIL (global interpreter lock) and really only useful if you're calling into a C/C++ lib that releases the lock while it does it's native things.
I've written a lot of Python in the last 15 years, and Ive written a ton of scripts that are faster than a well written identical C/C++/C# app, and was written in 1/10th of the time. It just depends on the use case.
Note that mypy/typed python fixes this: if you annotate a function as returning (or accepting) a Sequence/Collection/Iterable instead of a List, and attempt to mutate it, you'll get type errors. This is good practice anyway.
Swift runs on Linux and has done since it was made open source in 2015, a year after its initial announcement. Windows support is being worked on but, for the time being, it runs fine in WSL. No lock in.
I expect we may see some third-party, cross-platform reimplementations of SwiftUI very soon once the appropriate language features are set in stone and ship with the next version of Swift.
Otherwise, it's fairly healthy — lots of computational libraries, database, web, etc. Plus, Swift pretty much has direct compatibility with C and Python libraries without needing to wrap anything.
Could you please elaborate on this point?
Swift recently gained the features needed to call Python code. You can either use the Python module from Swift for Tensorflow or the PythonKit library, both of which allow calling Python code from within Swift.
The features used to allow calling Python from Swift can also be used for other dynamic languages, so there may be the ability to call Ruby and other languages' libraries in future as well.
>Kotlin: JVM lock-in
You know OpenJDK has been available for over a decade, right?
Mypy is a few years older than TypeScript, so, no.
The thing is, static typing is never the job by itself.
- script of 2,000 LOC to clean data => dynamic typing is fine => Python
- web service of 1,000,000 LOC => static typing seems a better choice => Rust, Go, Kotlin, Scala, Swift, C#, Java…
Function x does not specify a type, but returns SomeHelpfulTypeString. We recommend you use that.
Put it behind a flag and call it a day. Maybe even have a way of requesting type suggestions for functions during development
Yes, there are two things about introducing types that the author finds annoying. So don't do those two things, and use type annotations for all the rest.
The type annotation system and mypy are designed to give you benefits even if you don't fully annotate everything -- that's the starting point of nearly all existing python code bases.
class Node(typing.NamedTuple): # or dataclass
child: 'Node'
This error has been in there for several years at least. And it always bites me when I forget. It's particularly cryptic when the self-reference is many levels deep. I've even had the error message break in these cases where it prints out a line number from the wrong file. So, then you have to hunt to see where the self-reference is hiding.
from __future__ import annotations
from typing import Optional, NamedTuple
from dataclasses import dataclass
class Foo:
def __init__(self, foo: Foo = None) -> None:
self.child: Optional[Foo] = foo
Foo(Foo(Foo(None))) # Works
Foo(Foo(Foo(1))) # error: Argument 1 to "Foo" has incompatible type "int"; expected "Optional[Foo]"
@dataclass
class Bar:
child: Optional[Bar]
Bar(Bar(Bar(None))) # Works
Bar(Bar(Bar('bar'))) # error: Argument 1 to "Bar" has incompatible type "str"; expected "Optional[Bar]"
class Hep(NamedTuple): # error: Recursive types not fully supported yet, nested types replaced with "Any"
child: Optional[Hep]
Hep(Hep(Hep(None))) # Works
Hep(Hep(Hep({'a': 'b'}))) # error: Argument 1 to "Hep" has incompatible type "Dict[str, str]"; expected "Optional[Hep]" a = Foo(Foo(Foo(None)))
b = Bar(Bar(Bar(None)))
c = Hep(Hep(Hep(None)))
a.child.junk = 1
# error: Item "Foo" of "Optional[Foo]" has no attribute "junk"
# error: Item "None" of "Optional[Foo]" has no attribute "junk"
b.child.junk = 'a'
# error: Item "Bar" of "Optional[Bar]" has no attribute "junk"
# error: Item "None" of "Optional[Bar]" has no attribute "junk"
c.child.junk = 1.1
# error: Item "Hep" of "Optional[Hep]" has no attribute "junk"
# error: Item "None" of "Optional[Hep]" has no attribute "junk"
a.child = 1
# error: Incompatible types in assignment (expression has type "int", variable has type "Optional[Foo]")
b.child = 1
# error: Incompatible types in assignment (expression has type "int", variable has type "Optional[Bar]")
c.child = 1
# error: Property "child" defined in "Hep" is read-only
# error: Incompatible types in assignment (expression has type "int", variable has type "Optional[Hep]")
Looks like just plain class level attribute declaration also works:
class Node:
child: 'Node'
class NodeBase(type):
def __new__(cls, name, bases, attrs):
return super().__new__(cls, name, bases, attrs)
class Node(metaclass=NodeBase):
child: 'Node' # works
[0] https://github.com/python/cpython/blob/3.8/Lib/typing.py#L15...
Well, from [1], I'd say syntax is closer to C.
It does seem a little embarrassing that the annotations don't even support circular references.
Even if you just want to write a practical interpreter, and choose to gloss over the issues, they will still come back in some disguised form and either by requiring some sort of implementation kludge, or just by creating weird edge cases.
It doesn't.
Because of the syntax error.