Porting Python to GRUB: “Networking without an operating system”
lwn.net
lwn.net
The hardest part of porting Python in general was porting libffi to support the EFI calling convention, which required some very fiddly assembly. That libffi support allows us to call EFI protocols from Python without writing any C code.
The hardest part of implementing networking was handling the impedance mismatch between POSIX select and EFI asynchronous calls. POSIX select lets you ask "do any of these sockets have data to read", without actually reading the data; likewise for pending client connections to accept(). EFI expects you to provide a buffer to read the data and a callback for when the data gets read. So, the only way to ask for pending data is to buffer that data for the application to read later. This requires additional copies, limiting performance. (You might have heard of "zero-copy" networking; this is "several-copy" networking.)
Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard?
I understand that your goals were to implement as much of the Python stdlib's APIs so that programs can run unmodified, but could you envision Python applications that are coded against pyuv running on a platform that bridges pyuv calls directly to EFI without the stdlib as an intermediary?
No, it wouldn't. libuv depends on OS facilities like select, poll, or epoll, and then builds a high-performance event loop on top of them. We needed to implement those facilities themselves.
In theory, if we didn't care about Python's standard library modules, we could potentially port libuv to support the asynchronous callback mechanism used by EFI. However, we also wanted to write as much of this code as possible in Python, rather than C, to make it easier to maintain. That's why, for instance, we didn't implement the Berkeley sockets API in C to use the CPython socketmodule.c, but instead implemented our own _socket.py in Python directly on top of the efi module.
Python also lets us provide additional APIs to make scripting easy. For instance, we include the complete ACPICA interpreter, so you can evaluate an arbitrary ACPI method with arguments and process the result. And we have an FFI with EFI support, so you can locate and call an arbitrary EFI protocol.
The module that displays presentation slides using EFI_GRAPHICS_OUTPUT_PROTOCOL, on the other hand, was definitely "because we could". https://www.youtube.com/watch?v=bYQ_lq5dcvM#t=22m56s
Does your work have a high dependencies on Intel CPUs? ie, have your heard of it being used on other CPU architecture, eg ARM or PowerPC?
[1] I'm using networking as an example package, not knowing what other semi-optional packages could be used as an example.
46MB. However, that image includes three separate complete copies of all the compiled code (4.5-6MB each), so that it simultaneously boots on 32-bit BIOS, 32-bit EFI, and 64-bit EFI. It also includes the complete source code (24MB) needed to reproduce the image, to make it self-contained and redistributable without needing to have a separate distribution of source. And it includes .py files for all the BITS .pyc files (other than the Python standard library), to simplify debugging and hacking. A quick check suggests that you could pretty easily build an 8.4MB image, if you cared about the size.
> Is it modular, e.g. if you don't need networking[1] how hard is it to not include networking?
No. We haven't needed a smaller image, so we haven't worked on that. Typically, we boot it from a USB drive or hard drive, so storage space isn't a concern.
Networking, though, consists almost exclusively of Python code, and the compiled .pyc files for that don't take up much space; not a good target to start with if you wanted to reduce size.
Could you envision a (G)UI toolkit written in Python that could be used to write applications that run in EFI?
On a small scale, moving as much as possible from C to Python using ctypes. Much of the C code pre-dates us implementing ctypes support, so it provides bindings to things like ACPI via the Python C API. We'd like to re-write that using ctypes, which will result in far less C code and no manual management of things like reference counts of Python objects.
On a larger scale, we're outgrowing GRUB's menu system. It's been incredibly helpful to get started, but we'd like a more usable and flexible UI than GRUB menus based on dynamically generated GRUB cfg files. (Several of the menus consist of lines like "source (python)/test.cfg", where "(python)" is an in-memory GRUB filesystem implemented in Python, and Python code enumerates the test hierarchy at boot time to generate test.cfg.) In particular, we'd like to be able to have simple UI elements to select one of a set of configuration options and display the current value directly in the menu.
So yes, I can envision a UI toolkit written in Python. :)
The first few minutes of the talk got cut off; the conference organizers have the original footage and plan to re-cut and re-upload it at some point.
The talk from PyCon 2015 on the Python port itself is available as well: https://www.youtube.com/watch?v=bYQ_lq5dcvM
Anyway, thanks for the links!
Nginx is a pretty good example of this.
And luajit author, Mike Pall, is extremely skilled.
I'm running Lua on a DLX like processor implemented on a fpga, in this context Lua performs the same role as Basic did on many home computers in the 80's, a small IPL is setting up the machine and loading Lua on boot.
The target audience for this is BIOS developers and firmware engineers. Several existing development tools in that area already support Python, including Simics and the Intel ITP, so the target audience is used to it.
Lua also has several other impedance mismatches with the target audience, such as 1-based array indexing (remember, the target audience writes assembly and C).
While people often describe Lua as "easy to embed", personally, I don't find Lua's stack-based C API easy at all; in particular, the references to stack-based indexes that change when you push or pop the stack feels like working with stack-based parameters in assembly without a base pointer, where the offset of a given location can change at any time. I much prefer the Python C API.
Finally, Lua would require additional libraries to provide a useful environment, precisely because it can assume so little about its environment. For instance, we'd need to bundle one of the Lua socket libraries in addition to Lua itself, as well as an HTTP library. Python's "batteries included" standard library works nicely here.
Note that this port of Python is almost as old as grub's Lua; we've been working on BITS since ~2009, and it has had Python support since May 2011.
For reference, the Python port consists of less than 1000 lines of code, most of which provides compatibility implementations of standard C and POSIX functions that Python expects. (Plus another couple thousand lines implementing additional C Python modules like "_smp", "_acpi", and "_efi", but those are getting smaller as we move more code into Python using ctypes.)
In C, I make frequent use of "switch" statements.
Does CPython have a switch statment?
Note that Lua also does not have a switch statement: http://lua-users.org/wiki/SwitchStatement .
But let's consider Lua. Most of the Lua pogrammers I know also know C. How do they handle the so-called "impedence mismatches"? (Should this be plural? What else besides 1-based arrays?)
As far as I know Lua's target audience is, at least in part, C programmers. That's why it's made to be easy to embed in a C program.
You can't simply pull a few words out of the comment and assume it means there is no mismatch what C programmers expect and what Python offers. It doesn't mean that and wasn't meant to mean that.
You wrote "Lua's target audience is, at least in part, C programmers". That is also true of CPython. Bear in mind that these facts aren't that relevant. This code is not mean for the wider class of "C programmers". The relevant bit is "the target audience writes assembly and C". This is far different than "the target audience is assembly and C programmers", which appears to be what you think it says.
Instead, the target audience is the narrower class of people writing EFI binaries and working with the EFI protocol. As JoshTriplett further clarified, the "existing development tools in that area already support Python", but apparently not in Lua.
No problem. I get it.
Your other reasons for going with Python make sense, but this one surprises me. Isn't the Python C API's reference counting tedious? I've written bindings to C libraries for both Python and Lua. With Python, I always use something higher level than the Python C API, like Cython or ctypes. With Lua, I've cranked out lots of bindings using the Lua C Api directly.
I find a single call to PyArgs_ParseTuple at the top of a Python C entry point much nicer than referencing various values off the Lua stack as they move around.
That said, we're in the process of rewriting as many of those functions as possible from C into Python itself, using ctypes.
Or forth, or any number of alternatives.
You would get the same question with the names shuffled around whichever one you chose.
/snark:off
The biggest challenge would be adding equivalents of the assembly that implements SMP support (waking up other CPUs and putting them into a loop ready to run code on request).