Also, many statically-typed languages do not take advantage of the flexibility a good type-system can provide. For example, C, C++, Java.
Type systems like Haskell add a lot of value, but you have to really know the language to make good use of it. I mean, just IO handling requires planning ahead. Small dynamic languages allow one to keep simple things simple. There's value in that too.
I’m speculating here, but as a software engineer who’s dealt with many different languages, I (personally) find types to increase my productivity, because it reduces errors at runtime. This is a huge benefit, and because I’ve spent a lot of time learning the target language’s type system, it doesn’t come at the cost of productivity, and like I said, I find it to improve my downstream productivity.
I’ve read that scientists have had significant mistakes in their findings in papers due to bugs, which I know is anecdotal, but it seems like adopting typed languages would reduce those errors and increase confidence in studies.
I wonder if they were able to recognize the downstream benefits of types, if they would shrug them off as irrelevant to their work.
I definitely agree that if you're making a big custom model you should use a typed language. But indeed, that requires a lot of learning, so the investment has to be worth it. The problem is often that scientists don't program all day, so remembering language intricacies is hard. Dynamic languages that can have their whole syntax written on a postcard make things much easier from that pov.
Languages with algebraic data types also allow you to express state in ways that can make it easier to identify state changes, possibly making the missing increment more obvious.
Some languages will warn when variables are being used or modified in scopes where they are referenced but not used.
But, even with all this, yes bugs will still happen.
Which in at least some cases means ignoring bugs. Pretty much every system I've worked on that used a language that didn't to type checking (or was too forgiving about it) has had runtime bugs because of type incompatibility or type coercion issues.
Indeed, but simplicity is the name of the game for exploratory programming. It's difficult to explore using languages requiring a lot of ceremony.
But then, that's when the time and funding constraints come in and prevent rewriting in more type sound languages. Which is why we end up with buggy python (if speed not an issue) or buggy C (if speed is an issue).
There are alternate paths. For example, APL tries to be terse enough that you can have the whole thing on a single screen. That's another attempt at bug prevention.
For example, I do my data wrangling in J.
Some statically-typed languages don't require a lot of ceremony.
Let me rephrase that: Is it a bug if I don't know about it? Another way to say it, would you pay money to prevent a problem you're not (yet) aware of?
Saying the bugs "never happen" is basically saying "I always write bug-free code."
I feel languages that don't have type checking (or where it's sloppy), force developers perform semi-automated testing (i.e. unit tests to catch type issues) when they could have systematic, fully-automated testing (for type issues, at least)
That’s all I’m saying, the data analyst that has some edge case “bug” that never gets executed probably shouldn’t pay the upfront overhead cost of type checking.
The distressing example that nags at me here is how the headline feature of Apache Spark 2.0 is that they Greenspunned a dynamic typing mechanism into it. And, in doing so, realized improvements in ergonomics, performance, and memory consumption. Understanding why and how is an instructive lesson.
The most promising typing system for arbitrary data wrangling that I've seen so far is the one that's built into a Python package called Dagster. It seems reminiscent, as best I can tell, of dependent typing. (Although I haven't actually spent any time with a dependently typed language, so that could be a misunderstanding on my part.) The actual checks are done at run time, of course, but I don't see any particular reason, aside perhaps from the complexity involved, why a similar mechanism couldn't be implemented as a static type check.
In general, type annotations are useful if you know the business requirements beforehand. Data science is exploratory, where type annotations can be a form of premature abstraction. Languages with traits (e.g. Scala) might be a decent compromise.
However, it's possible to have a more lenient static type checker that only looks for cases where you're guaranteed to get an illegal operation at runtime. They answer the question "is this definitely wrong?"
For instance, if we have:
def f(x):
g(x)
h(x)
def g(x):
x.a()
def h(x):
x.b()
This sort of "this is clearly wrong" type checking is much less intrusive than the more common "I'm not positive this is correct" type checking.
If you're not doing whole-program analysis, then you may restrict the type search to the types imported into the transitive closure of the module and its dependencies. This makes type checking slightly more intrusive by sometimes forcing more module imports, but it's still much less intrusive than common type checkers.
If you're thinking about Erlang's Dialyzer, almost everyone I've heard talking about it says that it's difficult to understand and doesn't provide as much confidence as a regular type system would.
setattr(type(x),sys.argv[1],x.a)
setattr(type(x), 'b' if halts(sys.argv[1]) else 'c',x.a)With type checkers for dynamic languages, a lot of the arguments in favor of static languages disappear.
Python’s type annotations aren’t checked at runtime, though - they’re also entirely static. So you either have to:
- Write code that’s as pedantically correct as in a traditional statically-typed language (e.g. by using `mypy —-strict`), in which case you’d be better off using a language that supports AOT-compilation, or
- Be prepared for cases where the runtime type doesn’t match the annotation. This is unacceptable - I hate the idea of code that tells lies.
1) Scientists, hobbyists etc. can be super productive with Python while disregarding types.
2) Software developers can write serious applications with type checking.
Statically typed languages exclude group that benefit from 1).
Python should aim to be perfect for group 1. There are many other languages that will always be a better choice for group 2.
No, there isn't "always" a better choice for group 2.
- Most software isn't CPU bound, so that defies "always" in your argument
- You need libraries get things done
What language would you always pick for group 2? Or would you pick a different one for each task, spending an order of magnitude more time learning them?
Nonsense. It's very easy to use a runtime cast if and when you want to, even in Haskell. You're in control of exactly where typechecking doesn't happen.
In contrast, in a dynamic language with optional typechecking you have basically no guarantees. Even if 99% of your codebase is typechecked, you have no idea what errors might be lurking in the other 1% or how far they might propagate, since a type error can show up essentially arbitrarily far away from its actual cause. It's the worst of both worlds - you pay all the costs of a static type system but get hardly any of the benefits.
I'd say that the pool of python developers has been swamped by a tsunami of former java/c# developers who don't understand the first thing about what made python good and are demanding they get everything they were used to in their old languages in python.
It's a rather sad state of affairs, but it's been great for cashing out on a language I learned in my off time at highschool in the 00s, especially since I have my name as a contributor before I turned 18.
Python is glue.
If you build a stick bridge with just glue you sound like the type of person who ate most of it.
As part of the latter group, I don’t consider this change in focus to be a good thing.
It has much better integration with the gnu user space, and much like pythons early days if you know C you can write extremely performant code. The fact that I can write a DSL for every problem is a benefit I didn't know I wanted until I no longer had to monkey patch objects.
I have been using Lua as an alternative to Python. It's much closer to my ideal level of language complexity - and I think overall it's better designed than Python, although it's certainly not perfect. The implementation of the interpreter is significantly better as well.
Handful of companies? Small and medium businesses, according to most definitions, can have up to 100 employees, and they generally make up to 90% of the businesses in most countries. If these small and medium businesses are writing Python apps, it's far from inconceivable that they will at some point have tens of people working on a project.
You make it sounds like you need to be a company with 1 million employees in order to have a big project.
And I don't see how these changes impact the "millions (?)" using Python for small projects. Python type checking is optional.
It can be nice to use an optional type/spec system to help with that but not required
This type of quick'n'dirty throw-away coding would be a lot less convenient if Python would force a strong static type system on me.
If Python is used for "real projects" with more than a few thousand lines of code, worked on by a team, static typing totally makes sense. But there are a lot of everyday tasks where this just gets in the way.
But the optional Python annotations? I was skeptical, but I began to see the value in it, especially in interfaces/APIs for external consumption, or more complicated areas.
But yeah don't ask me to type hint a "private" function that's only a couple of lines long and used in specific cases
As a trainer, experience teaches me that it's also much easier for beginners.
Now, types are nice to have for bigger code bases, but that doesn't mean there is no place for a language with optional typing.
Well, sure, but that's a tradeoff for the developers of the language to consider. As a user, there are already plenty of statically typed languages to choose from with many person-hours put into just that.
It cuts both ways. Some days type advocates recognize why their type language is not their host language i.e., there is a such a thing as too much types.
If you believe types can express anything, and more over they are the best at expressing it then why do you need the host language, why not to program using just types.
There is no silver bullet. Types won't make your code bug free.
What programming language do you know that uses types to express most tests?
Example: Everyone in your system has a role. There are 3 kind of roles: User, Moderator, Admin
They have different attributes. All of them have a name, but only moderators and admins have dedicates areas of responsibility. And only moderators have a subset of at least one allowed permission (e.g. "edit posts" and/or "delete posts").
You can't really denote this in a way that the Java compiler understands it at the use site. E.g. an action is executed and you want to see if it is allowed. So you want to check the kind of role and if the attributes/permissions match the action or not. The compiler doesn't support this well at ala.
(As an aside, I think one solution in Java would be to use the command pattern and pass an object (ModeratorCommand, etc.) to whatever code is processing your actions, so it can type check the command.)
> Does e.g. Haskell support this?
Sure, and it is not certainly not the only language. E.g.:
data Role = User String | Moderator String [Area] NonEmpty Permission | Admin String [Area]
> (As an aside, I think one solution in Java would be to use the command pattern and pass an object (ModeratorCommand, etc.) to whatever code is processing your actions, so it can type check the command.)
The problem stays the same though: how do you know what kind of command it is that you just received? You cannot safely "match" on the command in the same way you can in other languages, because the compiler doesn't know that there are only exactly 3 types of commands.
> You cannot safely "match" on the command in the same way you can in other languages, because the compiler doesn't know that there are only exactly 3 types of commands.
Perhaps the visitor pattern? Command processing receives an object and calls its command method, which knows what permissions it has.
I guess the fact that I have to keep saying "patterns" is a hint that some flexibility is missing. Still, I'd rather have Java/C++ static types than nothing at all.