Grumpy: Go running Python
opensource.googleblog.com
opensource.googleblog.com
- It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it.
- If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what?
Python 2.7 is what's running at Google. Not really surprising they're looking at this considering the fast approaching end of (core dev) support for Python 2.7.
Write an interpreter in another language and programatically port modules to Go. Seems pretty sensible to me.
I'd prefer that all new Python tools that need to support 2.x also support 3.x. It's an additional development cost, but IMHO, a worthwhile investment in the future.
It's likely code using lots of C-extensions will continue with CPython2 and new code will be written in Grumpy (pure Python2).
You'd be surprised.
If anything, the numbers show the opposite. The vast majority of Python codebases, legacy or new, are 2.7 or older.
For 2.7 Pypy reported 419,227,040 downloads for 2016.
At the same time, for ALL 3.x versions combined (up to 3.6) there are just: ~52 million downloads.
That's 1/8th of the Python 2 downloads.
The message I would take from those statistics is that needing a fresh download of Pypy is less common among 3.x users than among 2.7 users, who apparently needed to reinstall from the web at least a few times a day during 2016.
Those are not downloads of PyPi, but of packages. It's not like "number of downloads == number of individual developers". Those are packages, including package updates. A single developer can download 50 deps across his codebase, and update them to later versions 2-3 times a year.
As if individual developers are the reason behind the bulk of the downloads. I wonder how many downloads Travis alone counts for?
Your hate of Python 3 in every discussion about it is frankly baffling.
Travis runs/tests user projects, so there's nothing about it that's especially partial to Python 2 over Python 3.
>Your hate of Python 3 in every discussion about it is frankly baffling.
Or, you know, my pragmatic assessment of its popularity.
That you'd even use the word "hate" (when in fact, I like Python 3 over 2.7, even if its mostly tame updates over what 2.7 offers) shows that you're probably too partisan. I was enthused with Python 3 even when it was only a vision called Python3K back in 2000-ish. My personal preference has nothing to do with whether I see more people using it or not.
The situation is not unlike the perennial "next year is when Linux dominates the desktop", which has been every year since 1999.
> The situation is not unlike the perennial "next year is when Linux dominates the desktop", which has been every year since 1999.
Your bias is showing, as it does in every comment section on this site regarding Python 3, as you make comment after comment about how inferior Python 3 is and how nobody is using it at all because your sample of 2 companies shows this and how it personally hurt your family or whatever. You don't stop. Either you hate it or you hate something else and use Python 3 as a vent.
>as you make comment after comment about how inferior Python 3 is
Actually I've never made any such comment. In fact, tame updates" means that IT IS an update over 2.x, only not that much as it could be. Which most people I've read agree, or at least agreed until the async stuff.
>and how nobody is using it at all because your sample of 2 companies shows this and how it personally hurt your family or whatever.
Notice how I never said that, but actually gave concrete numbers that place those using it in much less (up to 1/8 less) of those who use 2.x?
So why the lie? Less is not the same as "nobody at all", and doesn't fix by itself just because you really really wish more people used 3.
>You don't stop.
Yeah, I continue expressing my opinion and my argumentation. I should stop because you happen not to like it?
Please don't bring "the feelz" into technical and community discussions. It cheapens the argumentation. If anything, it's you who are biased: 80% of your submissions on HN are for Python stories.
One can acknowledge that D is way less popular than Golang or that Perl 6 failed to gain traction over 5, without hating Perl 6. Ditto for Python.
From what I've seen of his posts, he's only talking about the reality of the situation.. not "how it should be".
Go look at the stats on PyPi and other metrics. Python3 failed, there is a cutoff time for adoption. It's no different than the first 24 hours of a missing person report. You don't get eternity to see if something is going to pan out or not. We're past that point for Python3. It may survive as it's own (smaller) thing, but Python2 isn't going to die either and that's more assured than Python3's fate.
And coldtea is right, but we're not going to do your research for you. What I'm saying needed to be said to you, but you need to find better ways to contribute than just rebutting everyone who has something to say about Python3. Talking about how he hates "something else" and using Python3 as a vent is just ridiculous.
Occam's Razor would suggest that there are fewer Python 3 users
That's why packages like supervisor and graphite - which aren't libraries - are among the top downloads.
Those would exist for both 2.x and 3.x so it's not a differentiating factor.
Our legacy Python 2 build pipelines that we're actively moving off of hit PyPI far more often than our Py3 processes.
Maybe in your case, but from what I've seen, I seriously doubt use of Docker or Devpi makes any dent in newer Py3 codebase dependency downloads. Besides, tons of new codebases for greenfield projects are still done in 2.x Python.
Not sure how it is in scientific computing area, but for enterprise/web apps, any company that has legacy 2.x code and libs in production (which is most of them) will continue to write new parts (including new projects) in 2.7 for compatibility with their Python production setup.
3.x is either from companies that didn't already have significant 2.x Python code in production (generally newer companies that for some reason went with Python instead of Node or Go that the cool kids use) or new programmers that just get started and start with 3.x.
Grumpy is pretty much what most everyone would actually want out of a new Python and may have arrived just in time.
> The users never arrived
That's the way it used to be a few years ago - it changed a lot the last few years. Pretty much all libraries are ported and many new libraries are Python 3-only.
asyncio is nice.
All Python devs I personally know moved to Python 3. Porting is a lot less painful than it used to be.
[0]http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio...
I'm like really new to programming and I'm still just learning the basics, but I see this little addendum a lot from people who say everyone should be writing Python 3.
We're on Python 3 for all new stuff, and are migrating the old whenever we can.
And if Google can make an interpreter for 2, then sooner or later, one for 3 will pop up. Since Google made some restrictions on what Grumpy supports from python 2, I'm sure someone somewhere will be able to do the same stuff for three.
From a new features perspective, the other reply's Placeholder is fascinating. (I haven't looked into it thoroughly yet.)
I ask because I started to learn to code with Python 2 because that's what was preloaded on my system. Is one over the other a big enough deal at a beginner level that I should switch to 3 now? How much of a learning curve am I in for?
No, at a beginner level it's not, there are many guides that explain the differences at a beginner level, and you can go through those in a few hours at most, for example http://python-future.org/compatible_idioms.html
But, if you start working now on a Python 2 project and that project starts growing significantly, then it will be hard to convert the codebase. That's why you can see people saying that they didn't switch yet, it's not that they don't know Python 3, it's that upgrading large legacy code bases is hard (not only in Python).
My biggest program is like 100 lines of code maybe. So I will go ahead an switch now. But it's like 10pm where I'm at so here's hoping I don't play too much...
Now the rest of upgrades were a bit painful (I was using some functional programming stuff, httprequest libraries, etc.)
Python 3.5 with uvloop+sanic can be faster than node.js without any JIT:
The fact that Google has started this project to migrate away from Python to an AOT compiled language, shows where the performance wins are.
Here you go:
https://morepypy.blogspot.com/2011/08/pypy-is-faster-than-c-...
https://morepypy.blogspot.com/2011/02/pypy-faster-than-c-on-...
For more examples, just search "pypy faster than c".
Also, here is an article from the Python wiki about why speed doesn't really matter a lot of the time:
https://wiki.python.org/moin/PythonSpeed
And, my own two cents:
Speed is relative. Does every piece of code need to be as performant as possible? No. I would argue that, in most cases, speed of development is far more important than speed of execution. This is, of course, not true for things like drivers or statistical analysis.
Writing a web application? Speed isn't that important as the whole process is i/o bound anyways.
Writing a machine learning algorithm? Depends.
Web scraping? I/O bound, speed not really important.
Image processing? Speed matters at least a little bit.
Writing networking glue for distributed systems? Speed probably doesn't matter.
It's all relative. If it needs to be fast, it needs to be fast. Most things don't really need to be fast. For the things that don't need to be fast, why build them with C/C++/Rust/Go when you could spend half the time building them in Python/Ruby/js/etc?
I usually ignore it when talking about Python, because that is what the community does, by gathering around CPython.
> For the things that don't need to be fast, why build them with C/C++/Rust/Go when you could spend half the time building them in Python/Ruby/js/etc?
Because one can use languages like OCaml, Haskell, Lisp, Scheme, Racket, F#, C#,... thus having both the productivity of a REPL environment and the execution speed of native code.
I thought we were talking about Python implementations exclusively. My mistake.
>OCaml
I have no experience with OCaml, so I won't make any comments regarding it's efficacy.'
>Haskell
Well thought out language. I like the purity, but it's too academic for real world use outside of scientific computing. FP isn't for everyone, and my personally belief regarding it is that it is better used as a tool alongside other paradigms than all by itself as the only paradigm.
>Lisp
Lisp is useful for a lot of things. It's also not very popular for new projects as far as I've seen. There are also a ton of different versions, so I don't know if "Lisp" is really a good descriptor.
>Scheme
As far as I know Scheme is the defacto teaching language for most compSci programs.Or, at least it was for a long time. Once again, FP is not for everyone. A lot of people also dislike Lisp style syntax, myself included.
>Racket
Same issues as Scheme.
>F#
F# is a fantastic language. There's not really a whole heck of a lot to complain about other than the .NET implications. The only detriment relative to Python is F#'s much smaller ecosystem and community.
>C#
Once again, some people just don't like .NET stuff. A lot of people also see static typing to be a detriment in many use cases.
Relative to Python, these languages also share several other problems when it comes to real world application: lack of competent developers, stagnating ecosystems, lack of third party libraries, ecosystem lock in, and cross-platform comparability issues. In the case of languages like Haskell, they could even be considered "esoteric".
I'll give you that many of these languages are more "pure" or "logical" than Python. I'll even give you that most of them are designed much better than Python. None of that changes the fact that Python is overall easier to read, easier to learn, easier to write, has a better ecosystem, is platform and file system agnostic, has a very non-restrictive license, and is, overall, very pleasant to work with.
I'm not sure; as a Go developer, I kind of like the idea of having access to the Python library ecosystem from Go, without being forced to create an IPC bridge and building up the requisite release-management and deploy-time goop.
Plus, I'm just not a Python developer; in the case where the only library that exists to do something is written in Python, I'd much rather write Go that calls that Python library than Python that calls that Python library.
I'd like to see an example of this, because from the blog post I get the impression that this mostly allows accessing the Go ecosystem from Python, rather than the other way around. For example, how would Python classes be handled from Go?
Eg: python code (from blog post)
from __go__.net.http import ListenAndServe, RedirectHandler
handler = RedirectHandler('http://github.com/google/grumpy', 303)
ListenAndServe('127.0.0.1:8080', handler)Basically, we needed to support a large existing Python 2.7 codebase. See discussion here: https://github.com/google/grumpy/issues/1
> It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it.
There are restrictions. I'll update the README to make note of them. Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable.
> If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what?
It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated.
I'm guessing pretty much the entire AST module is a no-go?
What about stuff like literal_eval? Or even just monkeypatching with name.__dict__[param] = value ?
> It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated.
Would there be a succinct theoretical description of exactly how that's implemented anywhere? What about things like numpy arrays.
literal_eval could in principle be supported I think. name.__dict__[param] = value works as you'd expect:
$ make run
class A(object):
pass
a = A()
a.__dict__['foo'] = 'bar'
print a.foo
bar
Also, NumPy is implemented completely in C and Python, and makes extensive use of CPython extension hooks and knowledge of the CPython reference counting implementation, which is part of the reason why it is so hard to port to other implementations of Python.
from collections import namedtuple
class Foo(namedtuple("Foo", "a b c")):
@property
def sum(self):
return self.a + self.b + self.c
f = Foo(1,2,3)
print f.sumThanks so much for bringing it up.
If you like writing in functional style, namedtuples are much more natural than dict or classes, and more efficient to boot.
Namedtuples are a way to preserve the data unless the consuming code _really_ wants to change it, which is sometimes legitimate.
I'm not totally sold, as in some cases dictionaries or classes would add nice value. But namedtuples have a rigidity that makes you think twice before tampering with retrieved data.
1) You know before hand that the number of items won't be modified and the order matters since you are handling records. So it is a simple way of accomplishing that constraint.
2) Because they extend tuple they are inmutable too and therefore they don't store attributes per instance __dict__, field names are stored in the class so if you have tons of instances you save a lot of space.
Why creating a class if you just probably need a read-only interaction? But what about if you need some method? Then you can extend your namedtuple class and add the functionality you want. If for example you want to control the values of the fields when you are creating the namedtuple you can create your own namedtuple by overriding __new__. At that point it is worth it to take a look at https://pypi.python.org/pypi/recordclass.
namedtuple will have to be implemented differently. I think it can be accomplished by defining the class with type()? Maybe with a metaclass...
I've done it using more or less that method. The code is in the "coll" sub-package of my plib.stdlib project; the Python 2 version is here on bitbucket:
(2) Apparently setting a __dict__ key works; they could be implemented like that.
I managed to run into 2 trying to build a 5 line program :-)
$ cat t.py; ./tools/grumpc t.py > t.go;go build t.go;echo '----';./t
import sys
print sys.stdin.readline()
----
AttributeError: 'module' object has no attribute 'stdin'
$
$ cat t.py ;./tools/grumpc t.py
c = {}
top = sorted(c.items(), key=lambda (k,v): v)
Traceback (most recent call last):
File "./tools/grumpc", line 102, in <module>
sys.exit(main(parser.parse_args()))
File "./tools/grumpc", line 60, in main
visitor.visit(mod)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 302, in visit_Module
self._visit_each(node.body)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 632, in _visit_each
self.visit(node)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stin visit_Assign
with self.expr_visitor.visit(node.value) as value:
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 101, in visit_Call
values.append((util.go_str(k.arg), self.visit(k.value)))
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 246, in visit_Lambda
return self.visit_function_inline(func_node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 388, in visit_function_inline
func_visitor = block.FunctionBlockVisitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/block.py", line 432, in __init__
args = [a.id for a in node_args.args]
AttributeError: 'Tuple' object has no attribute 'id'1. Lambda tuple args are not yet supported -- I actually didn't know that was a thing :\ -- https://github.com/google/grumpy/issues/17
2. Even if that worked properly, sorted() is not yet implemented: https://github.com/google/grumpy/issues/16
def func((a,b)):
return b
mytuple = 1,2
print func(mytuple)
def func(t):
a,b = t
return b
This is probably the cleaner way to write that:
key=operator.itemgetter(1)Couldn't they supported with a slower runtime implementation? I mean I still love the idea and actually like the idea.
When I wrote "append from two threads .. as expected" I meant "two items will be added, which one first is unspecified", and the GIL certainly takes care of that.
I agree it does not protect you from your bad threaded code - but then, nothing short of STM does (and even STM doesn't guarantee starvation in the general sense - nothing can).
Nobody uses the features of Python which make it a dynamic language? Google must write some really weird Python if their compiler is that strict.
> getattr/setattr, or dynamically building classes with type()
Python has TONS of dynamicity besides those (eval and co), who are seldom used by anyone anyway....
If you think eval is what makes Python dynamic you're doing it wrong...
Grumpy doesn't even seem to try to implement that. That's a good thing. If you restrict Python a little, it's much easier to compile.
Isn't that more or less what RPython does? https://rpython.readthedocs.io/en/latest/architecture.html I mean, I know that starting with a full-fledged(?) Py27 codebase rules out _actually_ using RPython for the stated goals of Grumpy, but I think the two projects agree in principle and differ about the definition of "restricted" :-)
Frameworks do lots of such dynamic tricks in order to provide nice DSLs for building apps.
Does it really imply many restrictions? Common Lisp, for example, is probably more dynamic than Python and it's been a compiled language for ~20 years.
Common Lisp was designed for interpretation and compilation from day one. The first implementations from 1984/85 had already compilers.
> Common Lisp, for example, is probably more dynamic than Python
Some parts are more dynamic than Python, some not. For example everything that uses CLOS+MOP is probably more dynamic. Also some stuff one can do when using a Lisp interpreter may be more dynamic. CL is more static, where one uses non-extensible functions, type declarations, static compilation, inlining, ... The parts where a CL compiler achieves good runtime speed may not be very 'dynamic' anymore.
"Upgrade to Python3" is the usual defense to that, but it's not really practical for large companies with software such as YouTube completely written in Python 2.x.
Just tried this out on a reasonably complex project to see what it outputs. Looks like it only handles individual files and not any python imports in those files. So for now you have to manually convert each file in the project and put them into the correct location within the build/src/grumpy/lib directory to get your dependencies imported. Unless I missed something somewhere.. The documentation is a bit sparse.
Overall I think the project has a lot of potential and I'm hoping it continues to be actively developed to smooth out some of the rough edges.
I question the transpiler. I think I'd much rather prefer a solution like Jython.
Edited to add: The difference is that Jython doesn't covert python to JVM bytecode.
One advantage of an interpreter in general is that one important use case for Python is interactive scripting, as data scientists do.
Your assessment is right: the grumpc compiler takes a single Python file and spits out a Go package. Incidentally, this means you can import a Python module into Go code pretty easily.
I don't have a ready solution for building a large existing project but I'll write up a quick doc to outline the process. The trickiest bit is that the Python statement "import foo.bar" translates to a Go import: import "prefix/foo/bar". Currently prefix always points at the grumpc/lib directory so that's one way to integrate your code, but I need to make it more configurable.
class Test(object):
def __init__(self, value):
self.value = value
def method(self):
print(self.value)
class Test2(Test):
pass
t = Test("hello")
t.method()
Yeah, Grumpy does not currently support old-style classes. Since all of our code internally requires new-style classes, this was not a high priority feature. It is something that we'll get to.
All that stuff with "switch" seems to be to handle Python exceptions in a language that doesn't have exceptions. Maybe later, analysis can tell that some function can't raise an exception, and translated calls for such functions can be simpler.
two_32 = 4294967296
print(two_32 * two_32)
print(type(two_32 * two_32))
I was hoping for something more aggressive even, like compiling Python classes to Go structs so long as the program doesn't need the dynamic behavior. Alternatively, Grumpy could support declaring native Go types via some sort of pragma or a new `struct` keyword or some such, which would be treated like a normal Go object (rather than defining your Go objects in a separate Go package).
If you can identify the built-in types, that's most of the potential win; you get to do hardware arithmetic. If you represent integers as 64 bits and check for overflow, you probably don't need bignum promotion outside of crypto code.
But it gives different output; the print() prints a tuple whereas the function print() prints a newline.
Also compare print(1) and print(1,2) with and without the __future__ import.
How many interpreters are there now? And how many of them have even close to 100% compatibility with Python 2.7 or 3.N? Guido has lost control of the language, but has he's still officially the BDFL there's no real standardization body. His stubborn view on functional mechanisms have held the language back syntactically, breaking BC with Python 3 without fixing the language's fundamental problems... it really feels like Python is lost in the desert.
Which doesn't mean the language is dead, but it's rudderless. I think we were all hopeful when Guido joined Google that we'd see real direction for Python, but that obviously didn't happen.
Not that Python is dead, obviously - still lots of great projects are written in Python. But I don't like the language's future.
It is a good time to jump off the Python train in general, and I say that as someone invested in Python who loves it. If possible I'd recommend people reach for Go or Elixir depending on their needs or requirements.
I will admit I'm a little shocked how much of a failure Python3 adoption has been. I think if it had been Grumpy from the start it would've been a huge success. This is exactly what people want and Google should be commended for sharing this.
Here's to hoping Grumpy takes on a life of its own and is the new de facto Python.
I can appreciate you have that opinion, but I'll be livid if that's true - the last thing in the entire world I want is to do battle with dependencies and the very, very strict/opinionated Go build system.
If your idea is that all Py27 is just transpiled into Go and then jettisoned, that's fine, but keeping one foot in each world sounds terrible.
One thing is clear with all these new compilers/runtimes, you want to be writing Python2 syntax because that's where all the action is. I hope Grumpy succeeds and new features are added and becomes it's own ecosystem that plays nicely with Go code. These folks at Google have really done what Guido & Co should've done.
This is Python3 as most of us wanted it to be, it's worth rewriting all your code for... but you don't even have to do that. Valid Python2 is Grumpy already. I don't know what else I'd want. It compiles existing Python2 AND offers a legitimate upgrade from CPython at the same time.
As far as all of the lost C extensions? You won't need them with the performance that the Go runtime has. That's been the answer all this time, not maintaining C-extension compatibility.
They nailed this thing, it's the answer to "what's the future of Python?" that everyone has been wondering for the past 9 years.
> This is Python3 as most of us wanted it to be.
> Valid Python2 is Grumpy already.
> It compiles existing Python2 AND offers a legitimate upgrade from CPython at the same time.
> As far as all of the lost C extensions? You won't need them
None of the statements are true. You seem to be very confused what Grumpy can and can't do, and what the need of actual Python developers are.
The only benefit of Grumpy is speed (I don't think go "interop" counts). Now, that's a pretty big benefit for some, but comes with significant drawbacks and probably always will. Even though CPython is only the reference implementation, many clever people have worked to make it faster. Getting rid of the GIL is also very difficult. The easiest way to gain speed is to limit Python to a subset of features and then optimise for that. While this is a fair approach, hailing it as the future of Python is terribly misguided.
Grumpy proves Python2 is where the action is at. Everyone wanted a speed improvement with a new Python, that's the ultimate carrot.. instead Python3 was and still is in some ways slower than 2. Other than exec, eval and C-extensions, Python2 is valid Grumpy.
You didn't provide any reasoning or proof that my points, which were just reiterated, were false. If you're going to "port" anywhere from Python2, removing C extensions (which no language should have to be dependent upon anyway, so it's an improvement) and exec/eval usage is a bigger win than Python3.
The future of Python is what the users decide, not what the PSF decided. I recognize there's a lot of confusion and propaganda surrounding that. This is open source, not top-down control.
As a python 3 user everything seems fine on my end. Though 3 has its own new warts. They are smaller and more forgivable warts for now but its probably not a good sign.
I do agree the direction python is heading is not very interesting anymore but that doesn't mean its dead or useless now.
We also have third party vendors that only support Python 3 in experimental versions, and there not even recent versions (Bloomberg is a great example).
I really like Python3 features, but the pain of using them drives me towards using other languages. I hope Julia will be stable and mature enough soon so that I can dump Python all together. I really like Julia, but currently the changes in the languages are too fast and there are constantly incompatibilities with packages that don't update fast enough. But I'm reasonably certain that this will be fixed once they reach 1.0.
I'm sure Python is far from dead, but can imagine that Julia has the potential to kill it in many domains.
People will upgrade if they make py3 more appealing, something like a 20% speed boost would be nice.
Reference, for a non-Python dev who hasn't kept up with it?
Also, map and reduce were removed from the standard global namespace and into the functools module.
https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seL...
Basically, the language doesn't have a "spec" per-se. The language is whatever the defacto CPython implementation happens to do within it's giant eval loop.
Another great talk about CPython internals:
A) Not well optimised.
B) Touting features before the spec/standard.
EDIT: people really dislike that I said this, and I'm having trouble finding my original citation- it was on one of the many python books I own. Most likely "Learn Python The Hard Way" but I'll dig out the exact chapter where they compare pypy to cpython and mention that because cpython is the reference implementation it values code clarity over performance optimisation.
(edit: I'm just being polemic about your statement here. CPython is reasonably optimized within the constraints it currently has).
It makes sense that the reference implementation mirrors the same patterns than the language itself.
It explains why CPython can't improve on many things.
CPython is 25 years old -- people have been making it faster for a long time. Python 3.6, the latest release, has many performance improvements, cf. http://www.infoworld.com/article/3120952/application-develop...
It does[1]. And process of improving it is called PEP[2].
[2]: https://en.wikipedia.org/wiki/Python_(programming_language)#...
Python started as a one-man-band project and of course didn't have a specification.
C started as a one-man-band project and of course does have a specification.
JavaScript started as a one-man-band project and of course does have a specification.
Each of them also has a strong need for a specification, as there are many differing compilers and interpreters. There are a few for Python but are specialized, the CPython interpreter is good enough for 90% of cases.
No thanks. Written specs can always have interesting implications or undefined behaviour. Just because it's written in a more verbose language (English) doesn't mean it's less vague.
E.g. GGC is the de facto C spec for many. Code/platforms as spec makes more sense and is easier to maintain/update, with quicker iterations of language features (c.f. Ruby/Python to C++).
You can say it's imprecise or lack of ratification from one or many international organization(s), but you cannot say it doesn't exist. End of story.
It's interesting to see the same pain has now made caused the runtime itself to be implemented in Go.
It's a pity C extensions (often used in scientific computing) are not supported but Go does have support via CGO, so maybe some approach can be worked out to access C routines in the future.
Also, further signs that GC may not be the reason, is that D also has GC, but can link to C libraries somewhat easily (not sure about all cases or how far the ease goes).
Interesting, didn't know this (that Go code runs in an event loop). Is the reason something to do with goroutines and channels? something like, a routine gets info that data is available for it to read (on a channel, sent by another goroutine), via an event it receives?
Also, can you explain this point:
"which enables excellent I/O performance without kernel context-switches" ?
Before you can call any coroutine you first need to start an event loop and schedule something in it. This essentially enables the language to schedule another async function each time you use await.
Since Go by default always is async, before your main function is called, it sets up the even loop and then calls your main, which technically is also a coroutine. Your code appears to be sequential, but it is not executed that way.
The alternate stack structure is indeed one issue. The bigger one is the GC, though; the Go runtime needs to know which pointers it is responsible for freeing, and which are the responsibility of the C code.
That is not the bigger issue, and AFAIK already handled for C types.
The stack/calling conventions is the reason why cgo is "not go", cgo calls have significantly more overhead than just about every other FFI (the overhead of a cgo call is ~2 orders of magnitude more than a "native" go call, or was around the same time last year, that is you could perform ~100 no-op non-inlined native calls to a do-nothing function by the time you need for a single cgo call to the same).
Grumpy likely doesn't support the C extensions due to time, and complexity of having to actually emulate the GIL since Python does not have fine grained locking for structures. C extensions that work with Python data structures need to first hold the GIL.
It's because Python's C API is inherently non-thread safe. The API lacks passing an interpreter pointer as a parameter (as Lua's API does for example). So Python is forced to use a terrible thread local storage hack involving the Global Interpreter Lock to swap interpreter instances which is insanely inefficient and limits compute-bound programs to a single thread.
Python 3.x had a chance to fix the API and do away with the GIL once and for all, but inexplicably they did not. There was a misguided notion that C extensions between 2.x and 3.x could be interoperable.
In principle it's possible to implement something like JyNI (http://jyni.org/) or CPyExt (https://morepypy.blogspot.de/2010/04/using-cpython-extension...) to bridge the CPython and Grumpy APIs. In practice, marshalling data across the interface can be very expensive.
If this is good enough to run YouTube's python code already it's honestly super impressive. Well done.
There are a handful of C extensions for JSON, protobufs, etc that YouTube uses, but mostly they're small utility functions written by us to optimize particularly hot code paths.
(If the former, you could just update the code to use the standard Go JSON/etc packages..)
With what I learned about Go and concurrency, I would say that currently in Python, writing concurrent code is not very hard, and is as close to Go as you can get without actually just writing Go.
Now, you may be saying "but Python has the GIL, how can concurrency be easy in Python?" I'd say, you're definitely not wrong that the GIL is a problem, but it's not much of a problem for concurrency.
This goes back to the heart of Rob Pike's classic talk, "Concurrency Is Not Parallelism"[2]. To quote Wikipedia:
In computer science, concurrency is the decomposability property of a
program, algorithm, or problem into order-independent or partially-ordered
components or units.
In closing, concurrency in Python is pretty easy to work with, it just performs very poorly.
[1] - https://github.com/lelandbatey/defuse_division
[2] - https://blog.golang.org/concurrency-is-not-parallelism
https://github.com/rcarmo/python-utils/blob/master/taskkit.p...
Obviously, it wasn't amazingly performant. But it did help a lot for doing concurrent stuff, and I've been pondering re-doing it for asyncio.
My experience with Tcl teached me to stay away from languages that don't have either a JIT or AOT compiler on their reference implementation.
Just like people learned 8-bit BASIC and went on to do business applications and games on it. I went Z80 ASM instead.
Personally I would only use Python for shell scripting and advise for using Julia instead.
Of course, others see it differently.
> SciPy, which, you may not be aware, is basically C and Fortran code wrapped in Python API.
Which for me personally means, that I would rather C and Fortran directly or better yet, a C++, .NET or Java binding to them.
Being able to describe things with a syntax that looks almost like pseudocode and runs highly-optimized C/Fortran code to do heavy lifting has huge, huge advantages.
Also Java and .NET ecosystems are just a little bigger than Python.
LANPACK + BLAS = scipy
> Numpy/Scipy are only relevant to a minor set of computer users..
> ... Java and .NET ...
Being able to describe things with a syntax that looks almost like pseudocode and runs highly-optimized C/Fortran code to do heavy lifting has huge, huge advantages.
Thanks, I already knew that.
> Being able to describe things with a syntax that looks almost like pseudocode and runs highly-optimized C/Fortran code to do heavy lifting has huge, huge advantages.
Hence we are back at Scala, Clojure, F#, enjoying the respective AOT/JIT native code compilers, and integrating with that highly-optimized C/Fortran code.
Functional languages. The "year of Linux on the desktop" of programming languages. Also none of those look like pseudocode (Scala does if you ignore bits and squint).
Other sites have similar totals. But they are filtering for London/UK/Europe, so that might be a bias.
Do you have real examples here or is this just FUD? SciPy is not known for using absolute state of the art algorithms or perfectly optimized implementations. It can be pretty easy to improve on the naively implemented or legacy pieces of SciPy, e.g. http://tullo.ch/articles/python-vs-julia/
They also have the benefit of being able to push features into the Go core that they might need for this.
It will be silently abandoned in 18 months....
So the worst case here is if they never improve this past their own needs (which is a pretty limited subset). But if it's successful for them, and it's opensource, I could very easily see other people who run heavy stuff on Python contributing to it and helping it grow.
That's the thing with opensource, even if Google doesn't actively work on it, others can (if it has actual value and is useful to people).
A lot of big software companies do this nowadays. Google and Facebook both have a lot of purpose-built software, some of which gets released as open source, that meets their needs well but is hard to use for other purposes. I guess it's still strictly better than them not open-sourcing the code, but it's definitely an existence proof that just making something open source doesn't make magic happen.
I still think Unladen Swallow should have been based on V8, but as I recall that project had very strict compatibility goals that would have made a V8-based implementation impossible.
They port their python libraries so they can reference them from their Go code, and then module by module will rewrite it in pure Go.
Interesting - first time I have heard such an opinion. Why do you think it may be so? One reason I can think of, is they get more control over the languages they use.
Now with Python, they still seem to stick to Python 2.7 and don't show any effort to move. TensorFlow was released for Python 2.7 and only later Python 3 support was added. Grumpy is for Python 2.7 and first issue opened asking about Python 3 support was closed with change in documentation that only Python 2.7 is supported.
To me it seems like they wouldn't stick to Python 2.7 if they were planning on continuing using Python. It seems like it coincides with deprecating Python 2 in 2020.
I can see this happening for some core modules sure, but I think you've underestimated the work required to convert the sheer amount of Python code at Google. There is a tool inside Google which graphs the number of lines of each language in Piper; I don't think I can quote numbers from my time there but it would be no small feat, even for Google.
Look at AngularJs for a more recent example.
Add to this that Google has some of the worst versioning practices I've ever seen and you get a recipe for destruction.
http://www.jython.org/jythonbook/en/1.0/Concurrency.html
It can also handle Python's dynamic aspects.
In part it has exposed CPython "implementation quirks" that people were wittingly or otherwise taking advantage of. In other cases there doesn't seem to be obvious reasons for the differences and has required special-casing the python code to handle it.
It has been great with code written from scratch, specifically for it.
I am not sure, any implementation of Python will beat the single threaded performance of Cpython..
The main reason why Grumpy's slower for most single threaded benchmarks is that most Python workloads involve creating and freeing a bunch of small Python objects. In Go, these objects are garbage that need to be GC'd in a very general way. In CPython, there are free lists, arenas and other optimizations for allocating small (especially immutable) objects. And cleaning up garbage in CPython involves pushing unreferenced objects back onto the free lists for later reuse.
Right, I suppose I assumed that straightforward numerical-looking code would be translated to Go numerical code. Perhaps they just aren't that ambitious yet.
Compilers like this, from almost-Python to say C/C++, have existed for a while: Cython, Shedskin, Nuitka are some examples.
True Python runtimes fail the CPython test suite. They have some work to do!
There's a reason why the large companies often end up working on new runtimes/interpreters/compilers like HipHop for PHP, Hack, and so on, rather than working on the code bases written in those languages. It is very easy for it to not just be easier to leverage in at that level, but an order of magnitude or two easier. Or three.
Of course, if it started out as a "this seems like a cool project", that skews the "a is more efficient than b" ratio significantly.
Added up, it is easily more than 30s per line.
30s per line of current Python code, not compiler code.
And I'm just providing a number to put some numbers out there. 30s/line to convert a Python program to something fundamentally different like Go is probably an underestimate, yes, but then, if it's an underestimate that means the budget for the Python reimplementation is that much larger.
It also really helps the "subtle issues" when you control both the implementation and the code running on it; it means for every subtle issue discovered you have the option of either fixing the implementation or fixing the original code not to tickle the corner case. It's much harder when you only control one side or the other. It's not Google that will be having massive problems with corner cases.
And there's a lot more side benefit in a better multithreading Python runtime for Google for other Python code Google has (or hosts), whereas the benefits of a YouTube rewrite are more narrowly limited to YouTube.
- search (the various search pages)
- trending
- channels
- subscription
- video uploads
- history
- comments
- likes
- upload
- video editing (most of this is in browser, but still)
- livestream
- video analytics
- payment and ad management
- video comment review and moderation
- translation
- captioning
- video replication, multiscaling, caching, and resolution management based on network speed
- video recommendation
- cards, overlays, annotations, etc.
- music and sound effect search
- antispam
- dmca and copyright tooling
Not so much a feature of YouTube as it is a feature of HLS/Dash... but yes, it means you've gotta transcode the source video into multiple different bitrates.
Is this supposed to be some kind of no true Scotsman?
For a project the size of YouTube, that will be millions of dollars of engineering hours and weeks/months of lost productivity for an unknown gain and almost guaranteed bugs. It's a terrible value proposition so it's better to squeeze every last drop of performance out of the code base you have, which at the scale if this project includes paying engineer(s) to work on a completely new runtime.
I wrote a random sentence generator in Python several years ago. A bit later, I wrote my blog using Java EE. Early on, I had an idea: put that generator in Jython, and spit out a random sentence on every request. It's probably the one feature that I was OK to let go, should I switch platforms.
Since Oracle has only gotten more evil and Java more stagnant over the years (especially in light of TLS features), I've been thinking of possible alternatives. I've been intrigued by newer compiled languages, and it's come down to either Go or Rust, but I've yet to dig too far into them. I might have a winner.
Optimisation. This is a smart move, hard though. A compiler, written well allows the back end to improve the code. So the whole code base can improve with improved analysis.
"The biggest advantage is that interoperability with Go code becomes very powerful and straightforward: Grumpy programs can import Go packages just like Python modules!"
Extending Python (youtube codebase) with Go modules. That's interesting.
Not strictly true, http://doc.pypy.org/en/latest/stm.html. In general for the main project however, this is true.
It's clear that existing Python codebases will be maintained for the foreseeable future – there would be no reason to build this otherwise – but this may signify a shift away from Python for new extensions to the project, as this now makes it possible to integrate Go packages with relative ease.
> To solve this problem, we investigated a number of other Python runtimes. Each had trade-offs and none solved the concurrency problem without introducing other issues.
Not exactly an endorsement.
Sadly, the code was just dumped into a new Git repo, so no way to tell how many people contributed internally so far.
... but then I don't see Python going anywhere anytime soon. Didn't Microsoft just start a project to get Python's runtime to use CoreCLR's JIT?
There was an article, can't find it now, about an upcoming Python renaissance saying there may be an influx of new interpreters. There's PyPy, Microsoft's CoreCLR thing, now this, etc. It seems people really want to program in Python so there is an effort to make it faster.
*edit: found the article: https://lwn.net/Articles/691070/
I'm just afraid the surge of different compilers and interpreters will bring up plenty of issues in the medium term.
There is no formal spec of Python like there is for, eg, Javascript. (which is of course driven by multiple, VERY engaged adopters).
How long until subtle and not so subtle differences creep in between different implementations, leading to incompatabilities and a continuous fragmentation of the ecosystem?
Some work on that is discussed here. I would love a dropbox google colab (though also targeting 3.x :) )
One of the goals of open sourcing was to get feedback and work with outside folks so I'm definitely open to collaboration!
* It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces.
* Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)? Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)?
Of course being biased towards the JVM (since I know it so well) they could get really fast concurrency if they want with Jython today. Most of the Python tools already work with Jython (assuming 2.7).
With Jython you could always drop down into Java (or any other JVM lang) if you need more speed as well C for cpython (or even C from Java). It is unclear what you do with Grumpy with performance critical code. Can you interface with Go code or is the plan C?
Sorry, can't be very specific, but rewriting all the frontend code would take a lot more effort than writing a new Python runtime :)
> It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces.
We do iteratively rewrite components as well. We are pursuing multiple strategies.
> Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)?
Most async operations are performed out-of-process by other servers.
> Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)?
I'd love to support async and await at some point.
> Of course being biased towards the JVM (since I know it so well) they could get really fast concurrency if they want with Jython today. Most of the Python tools already work with Jython (assuming 2.7).
We did also do an evaluation of Jython but there were a number of technical issues that made it unsuitable for our codebase and workload. One such example is this longstanding issue: http://bugs.jython.org/issue527524. I just noticed the very recent update on that thread that implemented the workaround outlined in 2010 by Jim Baker. We tried that workaround and found we got a huge performance hit on affected code. There were a few other general performance problems as well but I can't recall all the details.
Please note I'm not at all bashing Jython, I think it's a great project with a sound design, it just wasn't right for us.
> With Jython you could always drop down into Java (or any other JVM lang) if you need more speed as well C for cpython (or even C from Java). It is unclear what you do with Grumpy with performance critical code. Can you interface with Go code or is the plan C?
You can interface with Go code directly, e.g. from the blog post:
from __go__.net.http import ListenAndServe, RedirectHandler
handler = RedirectHandler('http://github.com/google/grumpy', 303)
ListenAndServe('127.0.0.1:8080', handler)Here's a great article from a couple of years ago by Chris Seaton on this topic.
The biggest surprise for me is that the Go runtime would be a good fit for Python, performance wise, considering the very different object and dispatch model.
The post also mentions runtime reflection, which used to be painfully slow last time I used it. (Go 1.5, i think).
Has this improved in the latest releases?
If you know Go or are willing to learn about Go and reflection, you can learn a lot about how dynamic languages work under the hood by implementing:
func Add(interface{}, interface{}) interface{} { ... }
There's not a ton there. The next place I'd go is open issues: https://github.com/google/grumpy/issues
It looks like at the moment they're mostly random bug reports from people who have tried Grumpy since this announcement, rather than ones filed by people working on Grumpy since before it was made public. So that's a bit trickier.
The last place I'd look is then, the README: https://github.com/google/grumpy not a ton there about ways they wish to have people contribute.
At this point, what I'd do personally is open an issue asking how you can get involved; possibly by improving this documentation on how to get involved!
Anyway, that's what I'd do. Hope that helps!
to "what's good etiquette for contributing to open source projects?", I'd say: add as much information as possible to your PR, why it does what it does, how it does it, etc.
> To solve this problem, we investigated a number of other Python runtimes. Each had trade-offs and none solved the concurrency problem without introducing other issues.
For that matter, does Grumpy match CPython's Float behavior exactly?
ie.:
>>> 2.2 + 3.1
5.300000000000001
>>>Good? It's funny that a rabbit hole is the place you go to make obvious choices. This culture must seem strange to outsiders.
(Sure, none of my code will work, because it's all Python3 and usually uses C/asm, but it's still early and I'm hopefull.)
https://blog.heroku.com/see_python_see_python_go_go_python_g...
I suppose the easy concurrency and ability to inter-operate with Go libraries is really the driver for Go over that.
One way to leverage C's widespread availability and high-performance while side-stepping some of its deficiencies was simply to use it as a target for a different language. The Cfront C++ compiler which generated C code is probably the most famous example, but I recall that there were many others.
Maybe Go, like C, will make a fruitful target for other language implementations.
It was designed with the notion that all some people really need is just a better C.
For other people there's Swift, Rust, etc.
What I've long wanted is a better C++, not a better C. It would be hard for any such language to succeed today, because it will have to compete with Go, and Go has a fairly mature toolchain and widespread adoption and a lot of mindshare.
D literally is designed to be a better C++, and it hasn't really been a big success. Maybe that's because there really wasn't as big a demand for a better C++ as there was for a better C. On the other hand it may have been because Go sucked up a lot of the oxygen that D was going to need to succeed, and maybe that was because Go was a product of Google and D wasn't. (If this was my primary point, I'd make a more nuanced argument, though.)
I'm now wondering if the best way to get to a better C++ might be by piggy-backing on the Go toolchain. To get back to my original point, I'm wondering if Go might in fact be a good target for all sorts of better (for some definition) programming languages.
C++ has more features than any one person could ever need and it seems like the design philosophy is: anything that is possible in any language; should be added to C++. In reality it is a huge problem for productivity.
Maybe you're being glib, and that's fine. But improving build times, metaprogramming (which greatly simplifies many libraries), adding lazy ranges, more powerful type inference, better error messages - these are all improvements.
The fact is, better metaprogramming means c++ gets smaller, because you don't have to learn separate languages for run time vs compile time computation.
If you think the development of c++ in the past few years is anything short of amazing, you're quite mistaken.
As for D, I think it has numerous other issues, that don't apply to Rust.
Well if you really are tired of "better C" (in varying guises) then pull the fire alarm and go off-piste with OCaml or Haskell
It feels like you're mixing actual requirements (like "good performance" and "decent support for abstraction") here with things that are more like implementation details ("garbage collection", "statically compiled"). I don't really understand why you want "garbage collection", as such, though I could understand wanting some kind of increased productivity from not having to think about memory management. However, if your real requirement is as I describe it here, then I would still contend that Rust fits the bill.
It's always seemed like Google wrote Go to for the purpose of having a "faster Python"--where they were writing components in Python that weren't fast enough, they could write them in Go instead of having to resort to C++.
Rust is putting lot of effort in better marketing because they want much broader usage than bits of high-perf, low level code which cannot tolerate a GC pause.
Similarly Swift is putting effort on general Server side coding. But I think it will remain limited to server side bits of macOS/iOS applications etc.
Edit: for example, the fib benchmark they cite is cpu-bound. If the python code used multiprocessing, the performance would scale almost linearly with the number of processes.
Probably, but consider how much bare iron Google has lying around, much of it likely with few cores-per-CPU. Using it efficiently and not accelerating it's obsolescence is probably a priority for them.
BTW, does anyone have data that would suggest how long it does take an org at the scale of an Amazon, Google, or Facebook to entirely replace their HW? I assume that it isn't only through attrition, and that Google for example currently has no servers running that date back to Y2k, but I have no idea what the "half-life" of a server is at their scale.
[0] https://github.com/google/grumpy/commit/f60ee257db9d7996e3f8...
Google: Python -> Go
So has YouTube already migrated over to using Grumpy (and no longer running python in production)?
Around 2011 the beta releases of the static typing additions to Apache Groovy were called "grumpy" but the name was dropped after objections from the Grails crowd. I think the quality of the product is more important than the name, so Grumpy should do OK regardless if it's built and maintained properly.
that would be awesome.
As soon as something can see into the guts of the interpreter you have to maintain compatibility which is a pain/waste.
Worse than that is that view wasn't designed for multi-threading which is why the GIL exists. The C extensions were t designed to be multi-threaded because that wasn't a thing in Python so they're not safe. You either have to drop them, define a new interface layer that would be safe, or I suppose somehow sandbox their little view of the world but keep it coherent between threads.
If you have a codebase where you can make the choice to drop C extensions and you're trying to accelerate Python it seems like a very smart choice.
On the other hand, there are many C extensions which are just interop/wrappers for existing C code. I think no language is naive enough to think they can get away without C interop.
C extensions are actually pretty nice because you can wrap the C code into idiomatic Python. With ctypes, you have to e.g. maintain two structure definitions, which is very problematic for some codebases.
> In following PyPy (and other alternate Python discussions) it seems like eliminating the GIL and getting better performance out of Python, even in C, isn't that hard if you drop the C extensions.
AFAIK, many projects initially struggle to even achieve performance parity with CPython. The GIL isn't evil, it's a simple solution to a hard problem. But at this point, a complex solution to a hard problem is better if it's faster, and some people care more about speed than C interop (and vice versa).
What would be awesome is a Python runtime that could run with fine-grain locking until a C extension is loaded, and then continue with coarse locking. But the problem is still there's no upgrade path for C interop.
The only thing I can think of is using type annotations and something like Cython's cdef to write Python-implementation-independent C interop that doesn't suck as bad as ctypes. Then the Python rumtime could also lock the arguments at a very fine level while the function is being called.
yes yes I know tf now "does" 3 but we all know what Google really cares about.
"I'd like to support 3.x at some point" - trotterdylan.
Read: nice to have but when it gets down to brass tacks, 2.7 is where it's at for Google.
As always with Google...
What if you humbly contribute to open source projects rather than creating new stuffs, labelled with your own brand, controlled by your own engineers, and with your own design choices, however good they may be ?
Me too, especially as this was already several years after the YouTube acquisition.
Also interesting to note that potential internal customers were put off by having to upgrade to 2.6.1 in order to use unladen-swallow (presumably they got over that reticence at some point, as 2.7 is now standard).
Who would use 3 if you could have CPython2 for existing code and write new code in Grumpy Python? This is the dream language for me. Python on the Go runtime.
I'm hoping it becomes a permanent fork of Python2 that uses the Go runtime. That would really be great and exactly what they've got now.
This leaves us with more than 1 way to do things, like meta-class declaration.
There's a lot in Python 3 where changes were made to the syntax for 'clarity', but those worts weren't removed for any technical reason but because of the thought that since backwards compatability was being broken anyways, then we might as well get the most bang for our buck.
So two `builtin` modules, then?
What if someone does `sys.modules['builtin']`? Or any other kind of explicit string-based lookup?
How does pickle figure out which types to instantiate? There'd be a lot of types with same qualified names but different implementations with this approach...
It feels like the only way this would work reliably, is if you completely isolate the Py2 and Py3 universes. So if you e.g. pickle from Py2 code, it only looks at modules and types that Py2 universe knows, and vice versa.
But then what happens when code using the old library interacts with the new one (e.g. tries to pass objects around)? If that is prohibited, then you effectively still have two different languages, just with a single shared implementation - but no ability to gradually replace bits and pieces of code, for example, which would seem to be the biggest motivation for such a thing.
But the bigger problem other than lack of widescale user adoption is that the primary reason for Python3 (unicode) was botched. Go got this right, everything is a byte string and the assumed encoding is UTF8.