This may interest you. This is probably going to be the 'official' way to get what you're talking about.
To be more specific: getattr, operator overloading, descriptors, heterogeneous dicts, decorators, etc.
Type checking and metaprogramming are fundamentally at odds [1]. Dynamic languages like Python have more of a focus on the latter. They later added type checking, but it comes at the "cost" of ruling out the more idiomatic metaprogramming and reflection features. In other words, static typing makes your source code bigger.
Well, optional typing to some degree lets you have the best of both worlds -- you can skip type checking of the hard parts. But optional typing doesn't let you compile your program to make it faster -- you need a fully-typed program for that.
----
I'm doing something similar to mypyc with https://www.oilshell.org/ (I actually visited Dropbox and chatted with them about it back in the spring.)
The difference is that I'm compiling Oil's Python source to C++ rather than to Python-C extension modules. So it doesn't depend on the Python runtime. It's not done but it's working well so far, and it's given me a lot of appreciation for which dynamic features Python programs actually use! (both my own and others)
Also note that mypyc was used to speed up mypy, which is a type checker. A type checker is a very particular kind of program that's different than 99% of the use cases of Python. So success on speeding it up is super impressive but it's not clear it generalizes.
The same is true for Oil -- my translation work doesn't generalize to arbitrary Python programs. Lots of people have died on that hill because it's really hard. You have a hard tradeoff between the kinds of Python programs you can support and the speedup you can give them. There are 10-20 projects over the last 2 decades at various points along that spectrum. In addition to mypyc, Oil's strategy was also inspired by Shed Skin, which is an impressive but mostly dormant Python-to-C++ compiler.
----
So in short I would say the problem is that nobody will be able to agree on a subset. You will have a lot of different fragments of Python geared toward particular use cases.
But Python will very often be more appealing than any of those fragments because it has a bigger ecosystem. One thing that I've appreciated more and more while designing a language is how much the network effects and inertia matter. It's why we're still using C and C++ after almost 50 years. I'm sure every day there is still a lot more C++ written than Go, Rust, Swift, and D combined, etc.
Python has a similar network effect and it will be around basically forever in its current form. Software doesn't really get rewritten or reduced -- more stuff just gets added on top.
[1] I wrote some posts about that tradeoff here: http://www.oilshell.org/blog/tags.html?tag=metaprogramming#m...
I would say that dynamic metaprogramming is at odd with type checking (and optimizations, and in general understanding the behaviour of a program statically). But of course metaprogramming can be done perfectly fine in a statically typed language.
My blog posts give some more color on that, but also see:
https://discuss.ocaml.org/t/the-future-of-ppx/3766 (breakage)
https://words.steveklabnik.com/an-overview-of-macros-in-rust (breakage)
Also, OCaml has had at least 4 different metaprogramming systems -- ocamlmeta, camlp4, ppx, etc. Rust is getting its second one post-1.0 as AFAIU.
There are lots of open problems related to metaprogramming and type system design because they interact heavily.
This blog posts also hints at that: https://nim-lang.org/araq/v1.html
In contrast, in Lisp, metaprogramming is "just programming".
Also compare with C++ metaprogramming (whose syntax is certainly awful, although it has been continually improving), but works perfectly fine with its type system (in fact most metaprogramming in C++ is done via the type system).
On the other hand CPython also exposes the runtime internals to plugins and de facto that prevents the language from evolving and alternative implementations to gain a foothold, so the issue of exposing implementation details preventing language evolution is not restricted to static languages.
edit: the hard part of typing and metaprogramming is making sure your metaprograms are well typed, i.e. the generated program is guaranteed to typecheck. This is great, but it is not a strict requirement, if you are happy with syntax macros a la rust or unconstrained templates a la C++, there is no particular issue. Your genrated program will be still be typechecked at compile time, which is still better than having a runtime error because of a bad metaprogram.
It's exactly analogous to types being integrated in OCaml, Rust, etc. whereas in Python, JS, Ruby, and PHP static typing it's an "add-on".
Scala also has a newer system https://scalameta.org/
C++ has new dramatic new proposals 20+ years after templates, addressing really basic use cases:
https://herbsutter.com/2017/07/26/metaclasses-thoughts-on-ge...
The point is that no language has gotten it right so they keep introducing new systems and breaking old ones.
C++ templates also have the mistake where type checking is done after template expansion. That's why you get terrible error messages.
If you're not convinced, that's OK, but try watching the talk by Yaron Minsky here:
mypy and mypyc are interesting but their compile-time checks and optimizations are still hampered by Python's dynamic language semantics.
Smalltalk, for example you can completely change the structure of a class by sending a become: message.
What I think is missing is a bit of more PyPy love, and the Truffle and OpenJ9 Python support efforts.
I think the killer language will be typescript with access to both the python and JavaScript ecosystems. We'll see what that looks like.
And of course if something changes the syntax, better anonymous functions will be the absolute first thing I would look for...
I have not used TypeScript, but looking at it's documentation the syntax for type annotations look identical. Would you be willing to expand on why you think its approach is better / how it's different?
Try it a bit. it truly is enjoyable. Fifteen years of python and I'm still enjoying TypeScript more.
Mypy is limited by annotations having to be compatible python syntax.
This at least is being solved in Python: https://www.python.org/dev/peps/pep-0563/
> inline interfaces
Typed dicts in 3.8 look pretty similar at cursory glance https://www.python.org/dev/peps/pep-0589/
I'll definitely play around with it more!
// typescript
let t: {a: string[], b: number} | [string, string]
# python 3.8
from typing import TypedDict, Tuple, Union
class SomeTypedDict(TypedDict):
a: List[str]
b: Union[float, int]
t: Union[SomeTypedDict, Tuple[str, str]]
@dataclasses.dataclass # or @attr.s
class MyStruct:
a: List[str]
b: int|float
t: MyStruct|Tuple[str,str]
class MyStructTmpl(Protocol):
a: List[str]
b: int|float
def f(thing: MyStructTmpl|Tuple[str,str]) -> bool:
return True
f(MyStruct(a=['a'], b=2))
In JS having the typed-dict type makes sense because you're often working with arbitrary objects with who knows what attributes, but in python that isn't the case. There's fairly succinct and powerful tools (now, anyway) to define record types.
I'm also not onboard the "it's a code smell, it doesn't matter" train. IMO if python adopted the typescript typing syntax we'd all be better for it.
I also forgot to mention the atrocious typing syntax for functions. Once again, typescript is a lot more succint and readable.
I more or less agree with this, but then again, IMO you should be isolating the less controlled code behind a controlled api. And the marginal value of converting `_ConvertQueryResultDictToQueryResult(qr: Dict[str, Any]) -> QueryResult` to something that uses a typeddict instead (which may not be possible, since that function is probably generic) is low.
> I'm also not onboard the "it's a code smell, it doesn't matter" train.
Emphatically, this isn't what I'm saying. What I will say is that ergonomics encourage certain methods of development. From experience, I'm strongly against the pattern of using a dict as a weak struct. The best comparison I can give is tuple -> namedtuple -> attrs. Namedtuple has absolutely valid uses (when you need tuple semantics, usually for backwards compatibility). But people often use it for any record type, because it's easy and familiar. Dataclasses are usually better, and I'd be happier (and the average python code would be better) if the friction to add a dataclass was lower than the friction to add a namedtuple.
Similarly, if the friction to use a dict in place of an object is much lower, people will be encouraged to use dicts in place of objects. This isn't a good thing. That doesn't mean that we absolutely shouldn't try to improve ergonomics across the board, but I'm a strong believer that the language should make doing the right thing easier than doing the wrong thing, and this is often (but not always!) the wrong thing.
Maybe both should be in the global namespace...
Do you mean like https://www.python.org/dev/peps/pep-0604/#id17 ?
> Inspired by Scala language [5], this proposal adds operator __or__() in the root type. With this new operator, it is possible to write int | str in place of Union[int,str]
Naturally if the community finally gathers around PyPy, that might change.
In all 10 of the places it was adopted?
Compared to Python with the thousands of deployments "at banks and life science research labs" it's a no show....
(though, "competes" assumes Julia is getting anywhere close to Python's adoption in data science, but still)...
I agree with this!
> I don't think there is any sort of long-term future in anything "Python".
I disagree with this :)
I think Python has efficient IO concurrency built-in already with async, and I feel it is likely that it finds a way to work out CPU bound concurrency long-term, as projects like sub-shells with channel communication demonstrate.
Both seem rather hard to implement.
I predict that for CPU-intensive tasks, you'll keep using extensions in native code (like numpy or pytorch), or keep passing serialized objects through queues in multiprocessing setups.
What bizarro bubble do you live in?
Yes, and each new version actually adresses that, one step at a time. (see https://bugs.python.org/issue35813 for example).
Not to mention the language landscape is almost static at the top. Nobody's gonna came and take Python, Java, C, C++, JS, out in the next 10-15 years...
Only a huge self-blunder, like the Perl 5 -> 6 transition, and only at much more volatile time (when paradigms change, e.g. when web dev changed from CGI, Perl 5 had already lost the web framework scene to PHP, Rails, Django and the like even before losing its main niche back then - admin work) can do any serious damage to a top language...
Anyway, let's check in 5 years...
Do you believe that to be incorrect? I think you're probably underestimating how widely used those languages are in massive, "boring" companies around the world. Rust may be the new cool kid, and may even be the future, but the number of companies around the world that have adopted Rust for anything significant today is minuscule.
It has Groovy above Ruby.
Again, Java is everywhere, and many Java shops have added Groovy to their workflow where it makes sense. Ruby is barely used outside a small of number of tech companies.
Not in momentum, but there are tons of installations, and billions of lines of code in COBOL ever churning. If LOC was the main criterium (and not just a factor), and if COBOL projects were hosted in GitHub, most languages with be dwarfed by it.
And Groovy is semi-popular in the Java world, which is huge itself.
But as I said, TIOBE is very good in the top-10 languages, and for spotting new major contenders (by how they jump up spots).
It's not great for relative ranking of the longer tail of languages above the top-10 / top-20...
I think this is an extremely good idea. Python is horrible but forced on a huge number of developers because of its ecosystem ... I think a bridging layer from typescript to python could be built in a way similar to swift’s Python Interop — and I don’t think it would require any special language support ...
I think could actually make a better/easier to use/more robust design than Swift by requiring all interactions with the python interpreter from node be async.
This is a really interesting perspective to me. Coming from Python circles, I've heard too often how horrible JavaScript is as a language and how it's only used because the web has dictated it. Doing web development, I've used both, and generally am inclined to agree. I know TypeScript add some niceties on top of it, but it still is stuck with JavaScript baggage. My perspective has always been that Python is by far the better language, which is why people have written that eco-system in it despite the fact it doesn't have a built-in monopoly of the browser.
We build our employee database, and from there our IDM, from a singel XML file in a really shitty format + three txt files in even worse formats (they are single line output files from an old mainframe system predating sap). We used to do it in a rather complicated Microsoft SSIS workflow with a lot of C# services. All in all it’s a 30 minute nightly runtime. I recently replaced it with around 500 lines of Python and a 1-5 minute Runtime (sometimes at the beginning of a school year we’ll see changes to around 1000 positions).
Python eats the XML like it wasn’t shit. It takes things like terrible date formats, we’re talking the output of a SAP free-text box shitty, and ports then seamlessly into a SQL date field. This alone was a nightmare in C# and Python just does it.
Still, after two decades of strict types it feels dangerous.
The high-level-variant is a dynamic language with optional typing, which is good for scripting, fast prototyping, fast time-to-market, etc.
The low-level-variant is similar to the high-level-variant (same syntax, same features mostly, same documentation), but it has no garbage collector, typing is mandatory and it runs fast like C/C++/Rust. Compiled packages that are written in the low-level-variant can be used from the high-level-variant without additional effort at all. The tooling to achieve this comes with the language.
A language like this would be insane, IMHO.
At the same time, I’d love a stronger type system to avoid a bunch of the pitfalls that the dynamism of python has.
So count me in.
I don't know however if this approach could be extended to other domains - say making a web framework. Given, python classes let you do so much tinkering, any attempts to port existing code will probably need a lot of rewriting?
> I've been tracking nim, and would agree it's the most promising so far! I feel though that it's trying to be too flexible in many ways. Examples of this include allowing multiple different garbage collectors and encouraging heavy ast manipulation. I'm also afraid it is different enough to keep it from attracting a significant amount of developers from the Python community. Nonetheless, it's something I plan on using and contributing to, since it's the best option so far.
Though, now that another commenter pointed out mypyc: https://github.com/mypyc/mypyc I believe I'll invest my limited free-time in that project instead, as it will allow me to stay within the Python community and eco-system that I love so much.
Gives some good insight into where Nim is going in the future too.
> Do I have to rewrite my programs in RPython?
> No, and you shouldn’t try. First and foremost, RPython is a language designed for writing interpreters. It is a restricted subset of Python. If your program is not an interpreter but tries to do “real things”, like use any part of the standard Python library or any 3rd-party library, then it is not RPython to start with. You should only look at RPython if you try to write your own interpreter.
Cheers.
How's that a problem?
- I hate it's module system and package eco-system story. - I don't like its syntax. - I don't like its error handling. - I'd much prefer gradual typing. - I want to maintain the ability to use interactive interpreters. - I don't like the fact that instead of being community driven it is Google driven.
But, anecdotally, I see go being used as a second language to Python more than anything else and at an ever accelerating rate.
I agree, but if you for instance look at the TypeScript comparison sub-thread, you'll see that all the issues with both the syntax and implementation of the type-system are being aggressively resolved, and likely will all be so by 3.9.
> Good way to constrain the dynamism so performance could be improved
Couldn't agree more!
> environment
I find poetry a joy to use. If you want to bypass venvs all together, there's a lot of work to make that a reality, such as https://github.com/David-OConnor/pyflow.
> packaging
Python in 3.5 added complete zip app support, which has improved this dramatically from my perspective. Extended by things like shiv https://github.com/linkedin/shiv make it fairly complete.
> async/await
This is interesting to me. I prefer async/await in general, because it has become a standard across programming languages and I find it really easy to reason about. I also find channels to be too widely seen as a cure-all, when the only study so far has shown they actually led to an increase bug count. But I don't discount the value of real-parallelism, and am glad to see that Python has been pushing harder on that lately, with things like subshells that allow bypassing the GIL on a single thread.
I'm happy to hear that; hopefully the efforts really do address these issues well.
> I find poetry a joy to use. If you want to bypass venvs all together, there's a lot of work to make that a reality, such as https://github.com/David-OConnor/pyflow.
I'll have to check those out, but one inherent problem is that even if these tools really do solve my pain points, adopting them means I'm leaving my org on a relatively small island, isolated from the Python community. If these really are the holy grail, why isn't the broader Python community adopting them? Please don't take this as me looking for something wrong--whatever Python build tool I use, I'll eventually need support and there's a lot to be said for having a thriving community that has almost always run into my exact problem before.
> Python in 3.5 added complete zip app support, which has improved this dramatically from my perspective. Extended by things like shiv https://github.com/linkedin/shiv make it fairly complete.
We're currently using this via pex. It mostly works, but we still run into problems occassionally (system dependencies, for example). Figuring out how to integrate these tools into the broader build process is another problem to solve--we're using `pants` which supports pex out of the box, but we're running into lots of bugs or other problems. I'll keep an eye on shiv.
> This is interesting to me. I prefer async/await in general, because it has become a standard across programming languages and I find it really easy to reason about. I also find channels to be too widely seen as a cure-all, when the only study so far has shown they actually led to an increase bug count. But I don't discount the value of real-parallelism, and am glad to see that Python has been pushing harder on that lately, with things like subshells that allow bypassing the GIL on a single thread.
My biggest issues with async/await are
(1) every package needs an async variant (async boto, async docker, etc etc). We work around this by running them in a thread pool executor, and I think that works, but I don't know if I'm holding the GIL unnecessarily and causing performance issues (fundamentally difficult to diagnose). This is roughly the "what color is my function" problem.
(2) it's really easy to starve the event loop by calling into something that transitively makes a sync call or otherwise just does a lot of CPU-heavy work. We've run into both kinds in production and they've been really hard to troubleshoot (because the requests that time out often aren't the ones that are actually causing the problems).
(3) dynamic typing means it's super easy to forget to await things. Tests should catch this, but we find ourselves writing tests _just_ to catch this (e.g., we now write tests for entrypoints that _just_ `await lib_function(params)`; we would normally not write tests for such simple functions, but now we have to). Static typing is the right way to solve this and mypy does, but mypy has too many other issues (at the moment) for our org.
One substantial criticism of goroutines is that they're less safe than async/await because you need to make sure the code you're running is threadsafe. I appreciate this criticism, but I think it's the right tradeoff for Go's performance aspirations (another great high-performance alternative is Rust's borrow checker, but that's the wrong tradeoff for Go's developer productivity aspirations).
Only because they are so new. portray was built a few weeks ago and already has a thriving community building around it - but of course it's still a small drop of the whole ecosystem. Older tools I've built like isort, are now ingrained into the community: https://github.com/timothycrosley/isort, but that took years, even without major issues or complaints being present. It just takes people time to adopt new things.
However golang doesn't have that. So you get the danger of a dynamic language without the features that make powerful.
Very dynamic code shouldn’t be easy; the happy path should encourage clear, simple code. By encouraging people to stay on the happy path, their code is more performant, maintainable, etc and it keeps the average code quality quite high across the ecosystem.
1) Super easy. That way doing it right is trivial. 2) Impossiblely difficult so the only people who are doing it can be trusted to do it right.
To me go falls between those two. It’s real easy to say interface{} (indeed it is more difficult to make a non empty interface) but doing it in a way that is safe isn’t easy.
I don’t think expressive power is the point here. As they are both compleate languages. More it is an issue of what trade offs and comprises have been made.
These problems simply don't crop up in Go, or at least they're in a different ballpark in terms of frequency and severity. So yeah, Go lacks typesafe generics, but I'll make that tradeoff all day every day in exchange for the maintainability, performance, tooling, distribution, etc improvements that Go offers today. No contest.
C has void *, writing generic code using it is hell. Enough so that people went through a lot of trouble creating C++ and later Rust to escape it.
I'd say the type casting from interface{} to whatever you assume is in there qualifies as different.
Pretty much every single aspect of these languages is different from what I can see, the only thing they have in common is included batteries, the rest is growing popularity and consequences thereof.
What takes 3 lines in Python, takes 10-30 on Go.
Code reviewers glazing over copy-pasted boilerplate blocks can more easily lose track of the whole, and miss an error which is obvious when the whole is expressed in 10 lines.
There is some optimal range of expressive density for comfortable use by humans. APL or K likely above that level, and Go feels below it, not as low as COBOL, but still.
Boilerplate distracts from what is actually going on. I can generally identify code smells from the shape of python code (like, blur all the text so I can't read the words, and the shape of the blocks tells me everything I need), I can't do the same in go, because there's so much more indentation and visual stuff happening, and most of it (boilerplate error handling) isn't interesting.
Have a look at Haskell which goes to great lengths to eliminate boilerplate and I think you'll experience the opposite--Haskell becomes very difficult to read precisely because the code is so dense. Similarly, take the indentation, newlines, etc out of a JavaScript file or JSON blob (minify it, more or less) and see if it's more or less readable as a result. I think you'll find that visual structure is actually important.
> I think you'll find that visual structure is actually important.
I didn't say otherwise. What I did say is that go adds visual noise that isn't present in python. (and it is noise: the proposal to add try! shows that the error handling style is noisy. It can be basically entirely removed by an automated transformation). Actual pattern matching like rust has, or even what Google C++ has with StatusOr and [1] our nonsense RETURN_IF_ERROR macros are better than what go does, and just as explicit (actually often moreso, since its more difficult to forget an error condition)
[1]: https://github.com/protocolbuffers/protobuf/blob/d0f91c863ae...
Yeah, preference distributions are hard to assess. Either of us could be wrong.
> I didn't say otherwise. What I did say is that go adds visual noise that isn't present in python. (and it is noise: the proposal to add try! shows that the error handling style is noisy. It can be basically entirely removed by an automated transformation).
I’m glad we agree that terseness is not readability and visual structure is valuable. How do we meaningfully debate whether some boilerplate is noise or useful visual structure? Why is Python’s implicit propagation of errors elegant and beautiful visual structure while Go’s explicit error handling is ugly noise? Specifically how do we know that you aren’t prejudiced by your disproportionate experience with Python (even assuming my disproportionate experience with Python and preference for Go is an outlier)? What are the criteria?
All my projects now use poetry for the full build tooling and I love it. No setup.py needed just include any settings in the standard pyproject.toml file example: https://github.com/timothycrosley/portray/blob/master/pyproj..., which can be generated with poetry's help using poetry init.
> text-editor support
I feel like Python with type hints (for all their current flaws) does give you this exactly.
> dependency management
Again I think poetry solves the problems here very nicely
> documentation generator
Personally, I like portray better than anything in the Golang world for this https://timothycrosley.github.io/portray/ I may be biased since I wrote it.
I write a lot of Python tools so I'm genuinely curious because if there were unfilled needs I would want to address them as one of my 52 projects: https://timothycrosley.com/
We have yet to try poetry in our org. I'm hesitant to stray off the well-trodden path, but it might be worth a shot. Any idea about installing packages with system dependencies? Packages like `pygraphviz` (which depends on the `graphviz` or `graphviz-devel` system library) has always given us a lot of trouble, for example.
> I feel like Python with type hints (for all their current flaws) does give you this exactly.
I've noticed that some editors try to use these hints, but they seem to have a hard time in many cases loading the modules. It's possible that the editor extensions (e.g., VS Code) are just buggy, but it's still a problem. Further, they require that all of your dependencies have annotations or type stubs.
> Personally, I like portray better than anything in the Golang world for this https://timothycrosley.github.io/portray/ I may be biased since I wrote it.
I haven't tried portray.
The killer thing about Go's documentation generation is that it uses type annotations and exposes them in the generated documentation. This is critical because 95% of the reason I'm looking at documentation (especially in Python) is because I need to know the type signatures (and often Python docs omit types, or the types are wrong or vague--e.g., "the type is 'binary'" with no indication if that means a bytestring or a BytesIO or what). This is tablestakes for documentation systems in statically typed languages, but I have yet to find a Python tool that does this well. Further, `godoc.org` also generates links to types including across packages--this is _not_ tablestakes for statically typed languages--so you just have to click the type name and it will take you to the docs for other packages. Further, there is no CI needed to build/publish your documentation; `godoc.org` just needs access to your repo on github or elsewhere (you can run your own godoc.org inside your corporate firewall). Another nice-to-have feature is that documentation is just comments; there's no formal/obscure syntax a la sphinx.
> I write a lot of Python tools so I'm genuinely curious because if there were unfilled needs I would want to address them as one of my 52 projects: https://timothycrosley.com/
Cool. I'll take a look!
Which is positive mind you, but they would be better served by adopting PyPy.