A Python interpreter rewritten in Rust, that can run pip
rustpython.github.io
rustpython.github.io
What is the internal representation of a Python object? In particular how do you mutate a Python object while there may be other references to it - is it UnsafeCell, is there some checking?
How does the GC work? The GC needs to access and perhaps mutate all Python objects. In C++ a common technique is to make a linked list of scopes, stored in native stack frames; I'm not sure how that can be done in safe Rust.
> How are interpreters designed in Rust? What parts can be safe and what can't be?
Depends on the design. Different interpreters are different.
A quick glance at the source code seems to show unsafe in:
* A data structure called "boxvec"
* Some calls to unreachable_unchecked, get_unchecked, and the like. Wonder what the overheads were here.
* Some lock/mutex implementations
* A few unsafe functions in the jit. They use cranelift. Bet there's a bunch in there too. Usually this is the classic thing that needs unsafe in interpreters.
* The python object data structures; they do some stuff to have a smaller representation
* some FFI code in the standard library, calls directly to libc
> What is the internal representation of a Python object?
https://github.com/RustPython/RustPython/blob/master/vm/src/...
> How does the GC work?
Traditionally, Python uses refcounting + cycle detection. A quick glance at their issue tracker implies that as of last month, they had recounting but no cycle checks yet.
> In C++ a common technique is to make a linked list of scopes, stored in native stack frames; I'm not sure how that can be done in safe Rust.
Looks like they have a RefCell<Vec<FrameRef>>, instead.
I love the AST in XML approach.
Seems overkill for a language whose grammar is LL(1)?
Just curious, how did you know this?
> Python 3.9 uses a new parser, based on PEG instead of LL(1). The new parser’s performance is roughly comparable to that of the old parser, but the PEG formalism is more flexible than LL(1) when it comes to designing new language features. We’ll start using this flexibility in Python 3.10 and later.
More specifically, the grammar [0] is in EBNF form and is ELL(1). That means that if each EBNF production is converted to a set of BNF productions in an LL(1)-compliant way, the BNF grammar as a whole is LL(1). It seems that the Python tools themselves do not check this [1], but I have verified it myself.
However, as another commenter mentioned, the grammar doesn't exactly describe Python, but a superset of it. The compiler needs to perform further checks after parsing that "should" be part of the parser itself. A more advanced parser would allow these checks to be performed in the correct place - this is probably what RustPython does when using LR(1), and was one reason why CPython recently replaced its LL(1) parser with one based on PEG.
[0] https://docs.python.org/3.8/reference/grammar.html
[1] https://discuss.python.org/t/should-there-be-a-check-whether...
Solution?
async fn foo() {
}
fn main() {
let x = async { await foo() };
}
error: incorrect use of `await`
--> src/main.rs:6:13
|
6 | let x = async { await foo() };
| ^^^^^^^^^^^ help: `await` is a postfix operation: `foo().await`
And does it solve the infamous GIL (global interpreter lock) problem?
They need some form of locking around Python objects accessed from multiple threads, since Python has no ownership model. Do they add a lock around each individual object?
Skip down to "thread safety"
"Ownership models" are not a given. C++ has no "ownership model" and it doesn't lock around objects. Multi-threading users are just given the synchronization primitives and are expected to manage their own solutions.
The same is true in Python: https://docs.python.org/3/library/threading.html#lock-object... - in fact, if you're using objects simultaneously from multiple threads without using some sort of locking scheme, you're probably already in trouble...
The main thing that's keeping the GIL around is the reference-counting garbage collector. If your python implementation uses its own GC mechanism, it doesn't have this problem.
The GIL isn't some halting problem level thing. As you allude to, Python could definitely decide to go multithreaded and add in the requisite locks. They haven't.
I really wish that the PSF would
* Make a spec
* Break out the stdlib into a portable library, and by portable, something that can be shared across PyPy, Graal, Jython, etc.
* ??? yes
What's hard is doing it without dumpstering single thread performance.
I’ve asked myself this question (in a different context) and have concluded that, in general, developers are incredibly resistant to learning and using new languages.
For example: if they weren’t, then server-side JavaScript would never have become popular.
You make it sound like a personal psychological failing.
In real life, using the same language has business benefits: re-use of the same libraries, no duplicated business logic that needs to be used on both client/server, no need for training to the new language, no need for mental context switch due to the language change when working on the client or server part (aside from the essential domain knowledge of client vs server), less/common tooling infrastructure, and so on...
Looks like Rust is winning the catfight against Go, D, Julia, C++20.
Need to grab more popcorn.
Julia targets a different audience.
Go, for the most part, too.
C++20 is not going anywhere, and C++ is used more than ever.
D hasn't been a contender for 10+ years. It could technology wise, but it never gained traction.
All of Rust, Go, Julia, and C++ do well.
(Heck, C even continues to do well. Rust is not really eating into it).
It remains to be seen how the Mozilla situation (abhorent leadership, firings of Rust team people) will affect Rust.