What is the core of the Python programming language?
snarky.ca
snarky.ca
Skulpt compiles Python to JS in one shot, so it's not a million miles away from the architecture Brett envisages. (I gave a lightning talk about it here: https://anvil.works/blog/pycon-talk).
Things I've observed:
- "Ordinary code" requires remarkably little fidelity. Python has a very strong "simplicity culture", so most code avoids being clever. If you're creating an environment for users to program against (as we do in Anvil), it's astonishing how little of Python you need to support.
- "Be able to use popular Python libraries" is really, really hard. The ordinary user doesn't use all the esoteric stuff Brett mentions, but libraries do, left and right. Even porting PyPy's `datetime` implementation to Skulpt required filling out a lot of odd little corners of the object implementation.
We still have a pretty crummy subset of `unittest` because it's eye-achingly dynamic - for example, it really puts metaclasses through their paces. (Metaclasses are a perfect example of a powerful feature that really depends on details that descend directly from CPython: The API exposes the fact that variable scopes are literally `dict` objects, that classes are created dynamically at runtime, etc.)
- Finally (and most unfortunately): A lot of the strength of Python is in its ecosystem, and a lot of that ecosystem relies on native code. (Again, much more than you'd think. When porting stdlib modules from PyPy to Skulpt, we have often been disappointed by how many of them use native code - even in PyPy!)
The Python-to-C interface is really, really CPython-shaped. Anyone who wants to be compatible with numpy, scipy, etc, needs to literally emulate CPython's data structures, and then translate them to whatever faster/simpler thing they use internally. This emulation overhead is the main reason PyPy is slower than CPython on some workloads. (In Skulpt, we haven't even attempted this. It doesn't bother us much in Anvil, of course, as the server-side CPython is just a function call away, but for Brett's use case I'm guessing it would really hurt. One day we might try, with a JS-to-wasm bridge and doing the emulation in wasm, but it would be a hell of ajob.)
I was thinking the same thing... One can wonder how much value there is left if you build an alternative Python interpreter that can e.g. run in the browser, but cannot support C extensions.
I mean I like Python for what it is, but once you stray from the common/intended use cases, e.g. trying to embed the interpreter in a C++ program, skeletons fall out of closets around every corner.
For all it's merits, CPython feels like a house of cards, burdened by a legacy of bad engineering decisions it cannot get rid of anymore. It made me realize it's better to stop fighting it, use CPython as-is (which is great), and use other scripting languages for embedding, live-coding, web, etc. I very much prefer Python over JavaScript for example, but even then it wouldn't cross my mind to use some kind of Python-to-JS implementation for web frontend code, honestly.
At PyBay last year, I ended up with a knot of Python core devs talking about this problem, and talking about building a better, more portable API for native modules. The transition would be nasty (the 2->3 transition was..scarring), but it would solve this problem at a stroke.
> I very much prefer Python over JavaScript for example, but even then it wouldn't cross my mind
I would encourage you to at least check out Anvil (https://anvil.works). It's more than a Python->JS compiler; it's a reimagining of what the web as a platform could look like.
FWIW, I agree with you that it's a bad idea to swap out JS for Python at one layer, while keeping the rest of the stack intact. It's extra complexity, and it doesn't solve the actual problem with the Web, which is the sprawling complexity of the stack and all the weird frameworks that work around it (while themselves making the problem worse).
The Big Idea of Anvil is to replace all those different layers of abstraction with one big, coherent abstraction. That's one reason why we deliberately chose Python rather than JS, so that users wouldn't reflexively reach out and break that abstraction every time they Googled how to do something.
Pretty sure this exists already: https://github.com/pyhandle/hpy
https://anvil.works/open-source
(And yes, it even works with the Free plan)
I'd be concerned about investing too much effort in case my project strayed over one of the limits and I have to either pay £39 or kill the project. Whereas I would pay £5 without even thinking about it for access to all the libraries and perhaps a handful of emails etc etc.
Once I'm paying the £5, I can see my requirements potentially growing to reach £39/m but as a hobbiest I'd definitely never get there without stepping stones.
There's also stuff that surprises me around variable scoping, I imagine I'd get used to it but...
There are also just a bunch of foot guns I have to re-learn, like the comparisons often being bizarre, it's easy to work around all those when you're aware of them but there are a lot of those little eccentricities.
But like I said, I may not be entirely objective as I work almost exclusively in C++ and backend Python these days, so JavaScript is pretty far outside my comfort zone anyway.
A colleague of mine collected and compared a few Python-in-the-browser implementations - including Skulpt and Pyodide - with code samples and a description of the trade-offs.
You might find the write-up interesting: https://anvil.works/blog/python-in-the-browser-talk
The CPython interpreter is so slow that it makes sense to implement as much as possible in C. In the past, I've used this as an argument against slow interpreted languages - a faster implementation would not only run code faster, it would allow code to be written faster because more of it could be in a higher-level language.
However, I'm surprised that PyPy uses a lot of native code - not only does it have a fast JIT, but its whole raison d'etre is to be Python-written-in-Python - that's even the project's name!
When I was a young little programmer, the dream was to have a base layer of highly performant native code for handling the heavy lifting, tightly coupled to a very high level scripting language that wasn't really even trying to be performant. The thought being that the scripting language should be more focused on flexibility than performance, because, for the bits of the software that it handled, developer productivity was more important than raw performance.
Fast forward a year or two, and we are living the dream. It's Python, and, while I could nitpick implementation details and language features all day, the big picture is that it's pretty awesome. So awesome that, for the machine learning and data engineering work that I do, the most performant and most productive language I have to work in is Python. (Also R, which is a pretty similar technical story, but I realize that R lost the popularity contest, and I understand why.)
I guess it starts to feel to me like we're making the perfect the enemy of the good, here. We've got a well-established, productive, flexible, and easy-to-learn language where top-tier performance can generally be implemented as a library solution. Despite its many imperfections, that's pretty darned good. In my career to date, I haven't seen much cause to believe that I could realistically expect much better than that. I think that, when I do need better performance, I'd rather write a library than rewrite the language.
If this is “living the dream” then Go and other languages will blow your mind. You can have iteration that is at least as fast (especially when collaborating with other engineers) and performance that is 2-3 orders of magnitude better than Python. Not to mention a better packaging and deployment story.
The sad thing is Python could be much better, but it has to back out of some bad design decisions first.
Go's "packaging and deployment" is enough to rule it out for new projects for me.
So it's worth asking why that never took over the world as a counter to why python exploded.
It's tempting as a technology person to wish a "better python" had been the one to gain that sort of market impact. However, we have to at least ask the question if some of its technical flaws actually contributed to its success....
JavaScript would like a word.
EVERY interfacing with C sucks, even made on C!
Hopefully web assembly become good enough to change that...
In exchange, Starlark is GIL-free. This was the primary motivation to reimplement. It really is that easy if you don't need to be compatible.
I think Starlark is a good approximation of the core of Python. It is easy to implement. It DOES 100% look and feel like Python, and if you use it as Python-as-pseudo-code mode, you will never realize it is not Python. On the other hand, almost no Python library will run. This feels paradoxical, but that's what it is.
A Starlark interpreter can easily be compiled to webassembly and run in a browser. I did it here: http://laurent.le-brun.eu/starlark/
https://github.com/bazelbuild/bazel-skylib/commit/9948d5538b...
It's possible to `exec` bazel and have it work, with some trickery.
Which can be compiled to run on a standard platform, instead of the more common baremetal situation.
MicroPython does feature a REPL - though you can disable it so you can control UART in a more expected way.
A surprising amount of plain Python programs can run with the stripped down set.
Even though you won't find any of the referenced functions in sys. However, locals does work, as does builtins. You also get access to pip and a few other unexpected things like that.
And the sys.settrace tests are allowed to fail: [1]
PY_SYS_SETTRACE is a compile time option. It won't necessarily be a part of the build. I don't think it is by default on any of the ports at the moment, though I could be wrong.
Based on that, I'd call it an "unstable" part of the implementation at the moment.
[0] https://docs.micropython.org/en/latest/library/sys.html
[1] https://github.com/micropython/micropython/blob/master/tests...
That I couldn't find it enabled on any of the default builds is what made me conclude it might be unstable, especially as it is undocumented.
Fun fact: ABC was developed by Steven Pemberton (among others) who'd later led many W3C efforts, including the failed XHTML2 effort. I didn't get around to asking how he went from ABC to XForms (which is as idiosyncratic as it gets) when I saw him as a speaker a couple years ago ;) but anyway he's always been an inspiration.
If so, that is contradicted by the many people who have successfully used Python for many other purposes. That includes run-once scripts (where you might have used Perl in the past) where run time speed is not as important as speed of writing the code, but also certain types of code running in production. Like any other tool, you have to be aware of its strengths and weaknesses: maybe it's not the best choice for a code base running into 10s of KLOC or more, or for doing CPU-intensive work that can't be vectorised (e.g. multiplying a few large matrices in numpy is fine, whereas multiplying an extremely large number of very small matrices is maybe not fine). The fact it has weaknesses doesn't mean it should be written off for all tasks, even those where the weaknesses aren't so important, especially when it has strengths too (e.g. if Python would do for a particular task then using C++ blindly because it's "better" would be wasting developer time for no reason). It's often used for glue code where all the work is actually done in C libraries like deep learning, and it works very well for that: if 0.1% of your runtime is spent in the top-level glue instead of 0.01%, that's probably not a problem.
Also, this discussion is fairly off topic to begin with. The article really isn't about the strengths and weaknesses of the language. It's about what consitutes the language versus an implementation of it (CPython).
I know full well that Python is used a lot for "scripting" tasks (such as in yum/dnf and countless others), and has apparently a good standing in ML; in fact I'm using it daily (for example, I'm using BackInTime for backup). But whenever I come across eg. Python bindings in a package I want to compile, or Python used as part of a build step, I know I'm in for extra work since rarely does it work out of the box (relying on fragile "/usr/bin/env python" hashbangs, or having me to manoever around missing Python packages or versions, etc.). Saying this from the perspective of someone who has seen shell alternatives come and go, even Perl has, in practice, better backward-compat and has generally aged better.
But yeah, it's offtopic I guess.
Python is fairly popular for web backends, given the ecosystem built around WSGI, I would say that's one of its core strength, so async makes obvious sense.
Python's simplicity - which makes it appealing for beginners - is its core strength that had made it so widely adopted for so many different things.
I never learned perl, but nearly every time I try to do something complex in bash (more than a dozen lines or so containing control flow) I regret it and end up rewriting it in python. Then the next time comes around and I go "I really should get better at shell scripting, I'll do that," and bash my head against the wall for a while then end up going back to python again.
I'm similar with shell scripts - awkward at best. The longest one I've written is only a few hundred lines, but it was tough to write and test. It sits in the middle of a busy build pipeline, and while it does the job, I'm wary of making any changes.
For long-term maintenance, especially for other people that may need to keep it runnning, I'd trade some performance for a higher-level language like Python or JavaScript - the code and logic would be so much simpler.
Companies whose success relied writing v1 in Python and still using Python
- YouTube
- Dropbox
The real question would be: would the people who wrote these apps and especially the people who maintain them, write these apps in Python if they had to do it again?
Wow - I do think that it's a great teaching language although this definition could apply to Java, JavaScript, PHP and others. But "and not very much else" seems to me to be a crazy limited view of Python.
Many of the worlds most popular websites (including the one I work for) use Python successfully in production.
It's also now become the standard language (replacing R) for doing data science and machine learning.
I'm not in the science community, but my understanding is that is used heavily in scientific programming as well.
I'm sure there are plenty of other use cases I haven't mentioned here.
Python has a lot of drawbacks which make it a poor choice for use in production.
Duck typing means many classes of errors which could be caught by a compiler will make it into production, necessitating much more rigorous testing process (i.e. developer time)
Dependence on the system python interpreter and poor dependency management solutions make environment encapsulation at runtime a more serious problem in Python than other languages.
Python performs poorly and uses lots of memory compared to other languages, making it an expensive choice to build an application on top of.
Python's USP is that it has a shallow learning curve, which is why it has uptake in domains like science and ML which are populated by "non-programmers", but virtually every other property of the language makes it a poor choice for serious engineering work.
I think you'll get a lot of pushback for this sentence; not the first clause, but the second, since it contradicts the lived experiences of many.
But all sentiments come from somewhere. I'd be interested to understand how you arrived at that position?
I asked about that in the pypy-dev mailing list. Unfortunately I cannot find a link to that thread, but here are some answers (from Armin Rigo and Carl Friedrich Bolz):
> we (= the general PyPy developers) are not really interested in this direction [...]. There is notably the fact that RPython is not meant as a general language used to produce stand-alone C libraries. It's certainly possible to make one in theory, but cumbersome.
> One of the reasons why the RPython project does not care much about this use case anymore is that we had a way to make CPython extensions. It was called the "extension compiler". It turned out to be a) extremely annoying to implement b) not very useful in practice. Thus a totally frustrating experience overall.
> The reasons for slowness was mainly: when compiling RPython code to be a C extension module, reference counting is used to reclaim garbage. That is extremely slow. Therefore the modules never got the performance anybody expected.
> When people look at RPython, an obvious feature is that it is syntactically identical to Python. "RPython must be an easy language, given that it has got the syntax of Python, which is easy". This is a common misconception. In fact, pleasing the automatic type inference process can be difficult. It requires the programmer keeping in his head the global types of his whole program, and carefully writing code according to these implicit types. The process is much harder for newcomers, which don't have any written-down example to learn how to manipulate the types --- precisely because they are implicit.
> So this is the reason we are recommending against RPython now (and for many years now). Anybody who thinks RPython is as easy as Python is someone who will happily try out RPython and be burned alive by it.
Edit: I found one link to the thread: https://mail.python.org/pipermail/pypy-dev/2013-June/011503....
Edit: I also found a (closed, opinion based, obviously...) related StackOverflow question: https://stackoverflow.com/questions/17134479/why-do-people-s...
From the FAQ (https://rpython.readthedocs.io/en/latest/faq.html):
> First and foremost, RPython is a language designed for writing interpreters. It is a restricted subset of Python. If your program is not an interpreter but tries to do “real things”, like use any part of the standard Python library or any 3rd-party library, then it is not RPython to start with. You should only look at RPython if you try to write your own interpreter.
A related project (probably abandoned since a long time): https://code.google.com/archive/p/rpythonic/
Please do your best to support all the latest syntax though. Lack of support for some standard functions particularly hard to implement is Ok, lack of support for syntactic elements (e.g. type hints) is worse and, usually, easier to fix.
The issue is that it's dynamically generating methods. There has to be a way to get dynamically created code into python for this use case.
One great example is SBCL, a Common Lisp implementation which only has an compiler, but provides REPL, `eval`, and `compile` functions as per the CL spec.
For the question about the core of Python, I would consider that everything not tied to the external environment as Python. So `locals`, `eval`, `compile` must be provided to be called as a full-fledged Python. On the other hand, I won't include all of the stdlib functions in the definition of Python, like handling the file system or environment variables.
Think about node and the web browser - both are JS runtimes and they share the core language (like `this` or `with`, which provides similar capabilities of `locals`, and `eval`), but provide different stdlibs based on the environment. (Node provides web servers, file server interactions, etc... and the web browser provides APIs to manipulate the DOM, attach event handlers, etc...)
Nitpick: SBCL ships with an interpreter, but yes, it is turned off by default in favor of the native code compiler. See http://www.sbcl.org/manual/#Interpreter for details.
Arbitrary evaluation of source code at runtime, seems to me, exactly like the type of thing webassembly wants prevent. This limits a lot of the flexibility of a language like python (repl, eval, etc)
I feel like CPython isn't so much the problem as third party libraries that people have come to rely on that, under the hood, rely on architecture dependent code (numpy, scipy, psycopg, etc, etc). I would imaging that porting these is impossible.
I think a repl is fundamental to python more than any other language. I wouldn't say that about something else, but python, yes.
1. https://bitbucket.org/windel/ppci/src/default/ppci/lang/pyth...
https://boo-language.github.io
Boo's answer is "just syntax" -- libraries are .NET assemblies.
Early in my experience with Oracle's DBMS, I found the REPL handy to work out ideas. But a REPL is essentially a single line editor from which you can evaluate expressions/statements. As my ideas grew, the REPL became more of a hindrance. Having a full screen editor became more necessary.
Nowadays I work out all ideas within an IDE, firing up the REPL only for quick data checks. So my answer is that a REPL is helpful but not necessary.
It is not always single line, look at IDLE or even Firefox supports multiline REPL capabilities although it still needs some improvements (like not rerunning var declaration statements).
As some commenters said, I don't think Python without access to C/C++ projects would be worth right now giving how sloe CPython is. About 100% of machine learning, data science and general science Python code relies on C/C++ extensions.
Current CPython API and the need to keep compatibility reduces the capabilities of improving CPython.
Look at PEP-620 (https://www.python.org/dev/peps/pep-0620/) for more info on how python developers are addressing this.
Don't forget Fortran. A good portion of numpy is written in Fortran.
This seems to be the author's preferred approach, but I'm not sure it would provide significant performance improvements over simply compiling the existing Python interpreter to WebAssembly.
Compiling dynamically-typed Python to statically-typed WebAssembly would be similar to how Nuitka [0] compiles Python to C. This compiled code may be a little faster than interpreted Python, but it's still going to be quite slow - much slower in a browser than JIT-compiled JavaScript.
AFAIK the ideal way to maximize the performance of Python in the browser would be to JIT-compile Python code to WebAssembly. Are WebAssembly implementations able to support that?
WebAssembly is not JavaScript. So I assume iOS and Android also support running WebAssembly as part of an app?
You also need JS to interop back from WASM.
If the main goal is to be able to run Python code in a browser, another option is Brython[0]. According to the FAQ[1] it should work in all modern browsers, including mobile.
The Python documentation is pretty clear about (1) What is the Python language, (2) What is the Python stdlib (“Python” is either the first or the combination of the two), and (3) what are CPython implementation quirks. If you have all of #1, you have a Python implementation, #1 + #2 is, I guess, a Python distribution.
And, yes, most of the things the article questions are part of #1, and definitively part of the essentials of the language.
I think AST interpreted languages are becoming something of the past. Yes I'm aware there's bytecode in CPython. I like the idea of IL's as target platforms rather than 100 leaky interpreters that you have to manually bind your C code with if you want support ti support any one of them.
It really speaks more to the failure of C++ than anything about CPython. C++ is so much more than C with objects, and all that extra is what folks don’t want.
Manually-implemented dispatch / vtbls you mean?
What a bizarre way to phrase it. Shouldn't it be the other way round? Exceptions are just a "reinvention" of longjmp under a different name (yet identically noxious to program readability).
longjump is a specific family of processor op codes.
This is the same idea that while loops are implemented using goto.
- GTK (Gnome / GIMP)
- COM (Windows - though it is more intervined with C++)
- Quake / Doom and other old game engines though nobody uses C anymore today
Then why are you mentioning AST interpreters? Python is not an AST interpreter.