Python without an operating system | Hacker News Reader

Python without an operating system

lwn.net

219 pointsMacha73 comments

Next step: Python as an operating system. Like Chrome OS or Firefox OS, just with Python instead of a web engine. I'm a big enough Python fan that the idea really intrigues me.

(also, everytime I see an LWN article here, I keep meaning to subscribe, but I mostly read HN at work, and I usually forget by the time I get home... I feel really bad about that)

> (also, everytime I see an LWN article here, I keep meaning to subscribe, but I mostly read HN at work, and I usually forget by the time I get home... I feel really bad about that)

If you work anywhere even remotely related to Open Source, I'd recommend getting your employer to get a corporate subscription. It's one of the single most valuable news sources I read; it summarizes LKML and numerous other first-hand sources, as well as many conferences, summits, and other in-person events.

Lisp Machines, except in 2015. The beautiful phoenix returns.

Site's down now, but: https://mail.python.org/pipermail/python-announce-list/2004-...

Anyone want to take a stab at running it in QEMU?

http://uuu.cvs.sourceforge.net/viewvc/uuu/

Edit: http://dl.sourceforge.net/project/uuu/Existence/2003-09-21/e... should boot with qemu-system-i386

That would be interesting. This isn't quite freestanding Python, in that it's running on Grub so there is a lower layer, but it's still cool despite that.

Well, even Chrome OS and Firefox OS don't have their own kernels; they run on top of Linux.

If you were to build a Python OS, you'd probably start by writing PID 1 in Python. It would be a Python init, starting services written in Python, running gettys written in Python, and you'd log into a shell written in Python.

Hmm... I can actually see myself writing an init system in Python... (hell, I did exactly that at my last company.... I'd have to rewrite it from scratch, but that's no big deal...)

Great, I think I've found a new rabbit hole.

I wonder how much tweaking supervisord[1] would need to be suitable as a init replacement/for running as PID 1?

Incidentally, I also came across [2] -- which I wasn't aware of.

[1] http://supervisord.org/

[2] https://www.python.org/dev/peps/pep-3143/ https://pypi.python.org/pypi/python-daemon

Message me if you start this!

https://github.com/justincormack/ljsyscall is just that, but for Lua.

FFI to the kernel on Linux, NetBSD Rump kernels and even bare on Xen.

Every OS ever had a bootloader of some sort as far as I'm aware, in some cases you had to toggle them in through the front panel but I don't know of any OS that did not have a bootstrapping stage prepended.

The bootloader normally hands over the OS once it starts however, it doesn't continue running and acting as an interface to the hardware.

There's the bootloader, there's the bios and there is the operating system. It's perfectly legal to do bios calls from a running operating system (though there will be a price to pay for sure). The bootloader is usually discarded after the boot. The bios doesn't 'run' in any way either until you call it, think of it as a library that you can call.

Really would have to do something about the GIL in that case unless you want to be restricted to one core - or run several independent interpreters NUMA-style.

Yeah, the GIL is the biggest reason we haven't tried to run any Python code on non-boot CPUs; we just run tiny C functions that return information to the boot CPU.

Or a web engine made in Python.

https://en.wikipedia.org/wiki/HotJava

> I usually forget by the time I get home

Send yourself email reminders.

I really like Wunderlist. In the past few months, they seem to have fixed their syncing issues, so mobile and desktop are pretty seamless now. I keep tons of ideas in Wunderlist, and the search works pretty well if I just brain-dump some keywords into the todo item.

I also made a thing to suck all the Github repos you star into Wunderlist with their descriptions, since I always want to find something I starred in the past, and I've found Github's search lacking in this regard: https://github.com/ip2k/wunder-star

Or just use "at"[1] and "mail" (assuming you have access to a computer that is connected and on after you leave for work):

    $ at 8pm <<END
    > echo "Don't forget to subscribe to lwn!" \
    >   | mail -s "LWN reminder" personal-mail@example.com
    > END

[1] http://debian-handbook.info/browse/stable/sect.task-scheduli...

Next step: Python on bare metal ARM targets. Imagine loading Python interpreter to an uC, quickly hacking together some scripts and happily running some DIY project. `sys` and especially `os` packages are out, maybe some high level features and something clever has to be done about interrupts, but that would be totally awesome

That is already available: https://micropython.org I have the PyBoard but one can get it running on ST Discovery boards as well.

I seem to recall that the "Pi" in Raspberry Pi was originally "Py" and that the idea was that the system would boot straight into Python, similarly to how early personal computers booted into BASIC. This idea was later abandoned in favor of going with a full operating system.

That is how the pyboard works, with MicroPython https://micropython.org

There are a few real devices which boot up into interpreters, like the Espruino, which runs Javascript (!):

http://www.espruino.com/

Ugggh. I love the idea, and its great that JavaScript is everywhere, by why does it have to be JavaScript?!?!

There are quite a few devices that boot into Lua, see http://www.eluaproject.net/ plus the Tessel really does and just compiles to js, and there is Lua firmware for a wifi module https://github.com/nodemcu/nodemcu-firmware

The NodeMCU is an amazing piece of kit. I'd say it's the future for DIY "internet of things". I mean, a microcontroller with GPIO, spi, i2c etc. and built-in wifi hardware, running Lua, for $9. You can throw together "small screen showing live information from the internet" or "sensor uploading data to internet" for less than $20 total cost. The ecosystem is still very new, but it's growing really fast.

> for $9

$3 with free shipping

Link plz!

Ebay, search for "ESP-01".

This could lower the barrier for js-only developers/designers/beginners to get into more low level programming, or to get them to play with some hardware a bit.

Yeah, we considered JavaScript for BITS for about three seconds, before dismissing the idea of doing systems programming in a language with no integers.

Because JavaScript is everywhere.

There's also https://micropython.org - for even more embedded/python on microcontrollers. Maybe one can run python on the network card rom too?

>Maybe one can run python on the network card rom too? The network card ROM isn't microcontroller code, its just x86 code that runs on your main processor, called by the BIOS[1]. You could potentially hence have the OPs project run from the BootROM, if it would fit. It might be slightly tricky, as it runs as part of GRUB, which doesn't expect to find itself running from there.

Alternatively, the article mentions: > Environments like Mirage OS (and other "just enough operating systems") could also add Python using the BITS code without too much difficulty, he said.

You could therefore probably add this code to iPXE [2] without too much trouble to get python on your BootROM, though I'm unsure how much of the C standard library you would need to borrow or reimplement. By the looks of it, iPXE doesn't have threads or local storage support, so you would be pretty limited in usability without a lot of work.

[1] http://networkboot.org/fundamentals/ [2] http://ipxe.org

> You could potentially hence have the OPs project run from the BootROM

Just pointing out as the OP that I'm not the author of the project. I just came across the article on r/python

Mirage is built on top of the mini-os written in C that comes with the Xen sources.

Ah, of course. I should've said run python on your hard disk/cdrom... :-)

Author and presenter here; happy to answer any questions people might have.

Nice Work!

So maybe you can answer me this, how is memory management done in this environment? Are you free (heh) to do malloc and free or did you have to provide your own implementation? Is GRUB2 already in protected mode or there were some hacks needed on that area?

For memory management, GRUB2 fetches the memory map from the firmware (e820 under BIOS, BootServices::GetMemoryMap under EFI), and hands the usable memory regions to its memory allocator to allocate from. GRUB2's memory allocator then provides malloc, memalign, and free. (It also has some special functions for things like allocating low memory, needed for some system operations such as the CPU setup mentioned below.)

In the BIOS version, GRUB2 is entered in 16-bit real mode, and it transitions to 32-bit protected mode; it then has its own memory map. In the 32-bit and 64-bit EFI versions, the firmware transitions to 32-bit or 64-bit protected mode before calling GRUB2. (In all cases, there's no actual protection and the segment bases are all 0, though 64-bit mode does actually require a page table.)

However, GRUB2 normally only runs on the boot CPU ("BSP", bootstrap processor). We have to initialize the non-boot CPUs ("APs", application processors) ourselves. To do that, we allocate an aligned region of low memory, copy some bootstrap code and data into that memory, and send each processor an INIT (to reset them) and a startup inter-processor interrupt (SIPI) that forces them to jump to that address in low memory. We then transition them from 16-bit real mode to either 32-bit or 64-bit protected mode ourselves. (In the case of 64-bit protected mode, we set up paging by using the same page-table address that the firmware set up on the BSP, so that they all have the same memory map.) With that done, the AP can then run C code. Then we put the AP to sleep using mwait, waiting for work to do. When we want them to do something, we hand them a function pointer and a pointer-sized parameter.

Yeah, grub does enter protected mode for us, per the multiboot spec.

This is EXACTLY what I needed right now for a project. Does it already have a module to do the equivalent of an lspci? If not, where would I have to look to learn how to implement such a feature?

Note that PCI enumeration is a bit of an arcane art; if you want to know how to do it in practice on real systems with all their quirkiness, I'd suggest looking at how the Linux kernel enumerates PCI devices. (lspci in Linux userspace just walks the kernel-enumerated devices.) In particular, BITS doesn't have code to handle PCI-to-PCI bridges and enumerate the other side of them, or handle any quirks that crop up in real systems.

We have a very tiny Python pci module, which contains a brute-force bus-walk to find devices with a specific class; we're using that to find PCI-attached USB host controllers so that we can force them to "hand off" to the OS, which then causes the BIOS to stop polling them, which often causes the BIOS to stop doing frequent SMIs. (That allows us to tell if a BIOS's out-of-spec use of long-running SMI handlers is caused by USB handling.)

Simple PCI enumeration, though, is effectively just "walk every possible device, read the first four bytes of configuration space from function 0, if they're not 0xffffffff then they're vendor and product IDs, and then read the same from functions 1-7".

The presentation itself is on youtube[1], I'm glad I watched it, because I missed this sentence when reading the article at first:

>The presentation and demos were all done in the BITS environment, so, in reality, the whole presentation was a demo, he said to a round of applause

Pretty impressive!

1 https://www.youtube.com/watch?v=bYQ_lq5dcvM

Thanks! That was definitely the most fun part of giving the talk; several people told me afterward that that was the point where the stuff I'd been talking about suddenly became real (accompanied by various degrees of "wat"), because it wasn't just talking to some abstract "hardware and firmware", it was drawing on the screen, and that made it real. And then following that up with Python-generated fractals really got people thinking about the old PEEK and POKE days.

My first programming environment was C under DOS, where you could create a pointer to video memory at 0xA0000000 or 0xC0000000 and start scribbling on the screen. I learned from a book ( http://smile.amazon.com/Microsoft-C-Programming-Robert-Lafor... ) which had an appendix showing how to draw the Mandelbrot set directly to video memory, so this was really nostalgic for me.

There is similar "Erlang as an OS" project

http://nerves-project.org/

It is used for embedded systems but can work on a larger system too I suppose.

Basically it is just a kernel (for drivers) and on top (init) is Erlang VM.

The interesting aspect is an Erlang process is similar to a modern OS process -- has an isolated heap. So it kind of maps elegantly to an safe embedded multi-process OS.

Find the Erlang Factory talk by Frank Hunleth for more details.

Interesting, reminds me of HaLVM (Haskell on Xen) and Mirage OS (OCaml):

- https://github.com/galoisinc/halvm

- http://www.openmirage.org/

Erlang on Xen is extremely cool.

Fortran predates the first examples of what we now think of as operating systems. After turning on the machine you fed in a deck of cards that comprised the Fortran compiler.

http://biosbits.org/ and http://biosbits.org/downloads/bits-1154.zip - no Git repo yet.

Working on that.

Thanks for your hard work on this project - it's really really cool.

You know, how awesome would it be if a RPi booted up directly into Python in the same way old 8-bits booted up into BASIC? This could really help it better achieve the goal of being an educational instrument for young students.

I'm curious how it would be more educational to boot directly into Python instead of logging in and starting a Python REPL.

It's the same reason BASIC used to be burned into the system ROMs of old 8-bits instead of making you load it off of a disk.

Turn your question around, why would logging in and then starting Python be more educational than just booting directly into Python?

Because you have access to a web browser/document viewer/ text editor.

Because it takes longer to reboot than it does to restart a Python interpereter.

Because you can learn more than Python with it.

I don't think BASIC being loaded on old systems had anything to do with education and was really just a convenience factor. I think qbrass makes some valid points in the sibling comment.

First thing I thought about - though if you're looking to do this with a Raspberry Pi as a purely educational tool, it may be easier just to create the pure-python tool as a layer on top of linux.

Depends on what you're trying to do; both are useful. Booting to Python under Linux gives you an exploratory environment with access to operating system services, but not necessarily one that lets you poke directly at hardware, unless you put Python in the kernel. (And even then, the kernel and drivers are doing things behind your back.) BITS gives you direct access to hardware and firmware, and it's intentionally single-tasking, running only what you tell it to.

So if you want a comfortable Python environment with more services and the ability to use the full power of Linux, you would want to boot to Linux and run Python (possibly with some extra modules); if you want raw access to hardware and firmware, but without any OS (with all the advantages and disadvantages that implies), you want BITS.

Was that not what OLPC was about, before MS and other corporates got involved?

A small rugged laptop, booting Linux as the kernel, and a UI built entirely out of Python that the kids could modify at whim.

Holy cow. I've been looking for something like this. I'm currently trying to do board level testing on an embedded project with Python and the sysfs infrastructure in Linux. So far it's been decent but there are limitations to what I can do, such as testing all the ddr address lines.

Raw access to memory, MMIO, IO ports, registers, ACPI, and EFI should be pretty straightforward. If there's something you're trying to do that requires access to anything we don't expose in BITS, especially if it would require writing any new C code, please feel free to send us a patch, or just drop us a mail and we'll put it on the list to add.

If this was implemented with direct access to GPU as well then a new wave of scene.org creations may arrive!

Take a look at the video: https://www.youtube.com/watch?v=bYQ_lq5dcvM&t=26m15s

We have direct access to video memory in graphics mode. Have fun.

This seems to be a trend - python, before it erlang-on-metal. Is there a stable higher-level language runtime that runs on bare metal (that I can boot in a VM)? Or, better yet, something that compiles arbitrary LLVM code into .ISOs for deployment?

These modern "GRUB + interpreter" attempts reminds me of CP/CMS. CP was a hypervisor and CMS was a small single-user operating system that runs on top of it (scriptable in REXX).

After all these years people keep finding new things to do with Python. Amazing. Maybe I can take a stab at writing my own python os.

Once TCP and hopefully "requests" support is in there I can see this competing with iPXE - load and boot images from HTTP.

Does it mean that we gonna have a new smart generation of virus written in Python now? My my... ;-)

python's a great operating system, all it lacks is a decent programming language

pretty sweet, could this make things like bootloader menus easier to implement?

It did, yeah. We're generating GRUB menus directly from Python, so that we can dynamically construct menus of tests or lists of firmware tables.

GRUB's own scripting language is mostly up to the task of enumerating OSes and displaying boot options, though.

Reply on news.ycombinator.com