Building an Operating System for the Raspberry Pi | Hacker News Reader

Building an Operating System for the Raspberry Pi

jsandler18.github.io

331 pointsgregorymichael79 comments

If you're starting a new operating system project, you might consider using Zig[1] instead of C.

It does all the low level stuff that C does but it has some safety features as well as generics.

The standard library is usable in freestanding mode, even the data structures that allocate memory, because everything takes an allocator as a parameter.

Here are some examples:

* i386 OS: https://github.com/AndreaOrru/zen

* ARM (raspberry pi) OS: https://github.com/andrewrk/clashos

[1]: http://ziglang.org/

Why not Rust? Or, say, Ada?

Or go all the way and design a new language too!

Ah, the Knuthian approach. Why not go more than all the way and design an architecture too?

That would be to just use Assembly, given Mix.

Assembler is too high level these days. This would be translator from higher languages to hardware.

http://conal.net/papers/compiling-to-categories/

https://icfp17.sigplan.org/event/icfp-2017-papers-kami-a-pla...

Thanks for the reading material, quite interesting.

I think you are right, recently I saw an Arduino-like projects book, but using VHDL instead, which somehow pointed into this direction.

I don't know if this is my conservatism speaking or not, but one problem I see with this approach is that you wouldn't learn as much because more batteries are included. Having to implement more than just a kernel (like the standard library and userland) gives you more of a chance to learn intangibles like what is or isn't appropriate to include in an API or what programs will actually be useful given your overall design. Also, there could be the problem of not having as many resources available in the language you're writing your OS in. That's a huge deal for a less experienced programmer.

That all being said, Zig looks pretty okay. It would be a good candidate to write an OS if you had to because the standard library looks trivially portable so you could test the userland on a known working kernel before getting it working on your new kernel.

As an alternative to this you could use Nim[1] which compiles to C. If you, like myself, dislike C, Nim can be used as a "better C" (you can disable the Nim GC and forget about its standard library, then use Nim as a thin layer over C with modules, generics and other nice goodies)

1 - https://nim-lang.org/

I’ve been wanting to learn how to use a real programming language (I’m strictly front-end languages) and Nim looks great at first glance.

Other point of view, would be to consider that the language runtime is the OS being implemented.

There is ATS for the task, with all the C features plus proofs, quantified constraints, dependent types, linear types, interface for pluggable SAT solvers and so much more.

https://bluishcoder.co.nz/tags/ats/index.html

Apparently this is the shill-c-alternatives thread. Here's a short list:

* Rust

* D

* Zig

* C2

* Nim

* LISP

* Ante

* Clay

Please note: I think all of the above languages (with the exception of C2) are awesome and should be looked at.

* Myrddin

* Terra

You forgot

- Ada

- SPARK

- Modula-2

- Modula-2+

- Modula-3

- Oberon

- Oberon-2

- Oberon-07

- Component Pascal

- Active Oberon

- Dylan

- Object Pascal

- Sing#

- System C#

- Swift

- C++ (assuming C++17)

Then the older than C, or about the same age.

- ESPOL

- NEWP

- PL/I

- PL/M

- PL/S

- PL/8

- Mesa

- Mesa/Cedar

- Algol68-W

These are the most well known ones.

Plenty of material to learn for anyone that anyone with interest in systems programming and how it evolved since 1960.

I'm talking about legit languages people are proposing to replace C with that are actually any good (except C2, it was an afterthought and I regret it because it was out of place and unlike the other languages there purposeless). Ada and perhaps C++17 is the only language people are actually recommending to replace C.

Yeah, given the current market situation, you are right.

However, there are quite a few lessons on the manuals, papers and sample code from those languages, to learn about other approaches of doing systems programing.

Lessons that one can bring back to life.

For example, are you aware that ESPOL, the systems programming language for the Burroughs B5500 in 1961, about 10 years before C was created, already had the concept of UNSAFE code blocks?

Burroughs B5500 is still sold nowadays as Unisys ClearPath mainframe.

Or that IBM implemented their research OS and compiler toolchain for their very first RISC processor in PL/8, a safe systems programming language, using a compiler architecture similar to how LLVM looks like?

IBM mainframes still use PL/I variants, although new systems code is a mix of C and C++.

So plenty of ideas there to learn, even if no one is going to start writing new code in those languages.

> Ada and perhaps C++17 is the only language people are actually recommending to replace C.

I would still add Swift to that list, even if that turns out to be just like Objective-C, mainly specific to Apple platforms. Given Apple's roadmap for it.

Swift? I know Apple says they intend it to be usable for OS development, but I would want to see some real OS work (either from them or from others) before being convinced it is a real contender (or is there, and did I miss it?)

Officially launchd and the Dock are now written in Swift (disclosed at WWDC 2017), so even though they aren't drivers or something, they are already a bit lower level than just apps.

Swift 4.0 got all the Unsafe* APIs stabilized, maybe they would be using it a bit more.

In any case, Apple also suffers an Objective-C vs Swift, just like Microsoft used to have C# (Longhorn, Singularity, Midori) vs C++.

Which means, even if it possible to use a certain language from technology point of view, many political wars have to be won as well.

Swift is more applications programming than OS or systems. It does have those capabilities, but no one is using it for that and it's not really the intent.

However, to the extent that swift (and obj-c!) are used for systems programming, those programs are useable outside of apple platforms.

It is definitely the intent of Apple, explicitly mentioned on their developer documentation as C and Objective-C replacement.

"Swift is a successor to both the C and Objective-C languages."

https://developer.apple.com/swift/

As mentioned in another comment, most of the time the issue is political, not technical.

LISP Is Systems Programming

or, if you want,

LISP: Loses In Systems Programming

LISP machines?

When I started this, I considered writing it I rust. However, I really wanted this to be the tutorial I would have wanted to read several years ago when I first developed an interest in OS, and a "modern" system language like zig or rust would have hurt that goal.

Because real programmers don't use such tools?

What? No. Because way more people know C than rust. I wanted to be able to reach a wider audience. I know I would have looked elsewhere if I saw found this but I didn't know the language.

Oh shoot. this is mine! If you have any questions or suggestions, feel free to ask me!

Typos I see from quickly poking around:

Part 1: "tar -xf tar -xf gcc-arm-none-eabi-X-XXXX-XX-update-linux.tar.bz2" should probably lose one of the "tar -xf"s.

Part 6: "unfmilaiar"

Part 8: "Proccess"

In what order should the extra information pages be read in relation to the tutorial parts?

The extra information is all linked to from the main parts. The reason it's also on the main page is so anyone looking for that info independently of this particular tutorial might find it more easily.

this is excellent, i've been thinking about this as well recently. my goal has been to create a small device/appliance using raspberry pi. i tried out lots of operating systems: raspbian, ubuntu-mate, ubuntu core, windows iot, fedora, etc. i needed a lightweight x windows environment, a decent package manager, and modern linux kernel, overall none of these really seemed to match my needs. basically raspbian was too heavyweight, it was a desktop environment with way too many applications like libre office, etc that i just didn't need. ubuntu core required the use of snaps which are based on docker, i'm hosting an electron app so that's just overkill, i'm thinking in terms of preserving memory, too since there's only 1g on the rpi 3 model b so everything has to be optimized. windows iot requires a windows license, a definite non-starter for me. i even looked at chromium and android, but those didn't seem to be a good match because i want to host a golang service and i don't have a touchscreen. i even looked at fuchsia, but that seems too experimental.

i even considered doing my own os, but ultimately my goal is to build an appliance so that would be a major distraction. dietpi seems good, i will probably pursue that, but thanks for writing the article.

Why can't you start with Raspbian and uninstall most of the apps?

that's something i considered, but i also wanted to take the opportunity to learn more about creating a tight knit distribution. i could see a lot of appliances coming out of these rpi like devices, there definitely needs to be a tight distribution purpose built for people who want to build appliances similar to mine. i've also always had this task burning in my list of things to do to do an lfs type project.

my device connects to any hdmi enabled tv and provides a digital scoreboard for sports, basically i'm running a chromium browser (via electron) and autostarting that when the device starts. the device is then controlled by an ios app. i ended up using ubuntu-mate, but i don't think i even need gnome, maybe something along the lines of windowmaker, flux, or lightdm would do. i need to save microsd space so that i can have users upload media like images and video, and ubuntu mate really hogs up 4G and above which forces me to have to supply a larger microsd card which ups the total cost.

An in-between option might be to use https://github.com/drtyhlpr/rpi23-gen-image -- it generates a Debian distribution for you with (more or less) only the packages you specify.

The most recent image I built ended up being around 700MB, but I included some stuff you may not need, so you might be able to get that smaller.

There's Raspian Lite, which is pretty much exactly that. You can just add a window manager onto it.

will give it a try, i think raspbian is sort of targeted at the education market for students that need a full blown desktop computer for various things. the fact that it's running debian as a base is great. how does this differ from dietpi?

DietPi is smaller, but I'm not sure exactly what it's missing that Raspbian Lite has. I've looked into it before, but I don't remember the answer. They're both Debian-based.

This has a list of DietPi's pre-installed applications and the packages that it has available for installation: http://dietpi.com/phpbb/viewtopic.php?f=8&t=5#p5

i noticed dietpi has a great list about the various hardware sbc's and their performance ratings for things like file system storage, etc. time to also check out the asus tinker board or odroid c2.

You might want to check into Tiny Core Linux.

what is the package manager used by tiny core? i think this is what's used by boot2docker

If I remember correctly, TCL uses a very simple overlay system based on squashfs. Most (all?) of the tooling and package management stuff is also written in bash.

I used this tooling: https://github.com/ali1234/rpi-ramdisk

To build a tiny fine-tuned Raspbian for "embedded" Python 3 (a project I dub PiratePython). It uses multistrap/qemu and worked well for me. YMMV.

i suppose this is a one-off project for you or are you creating a device for distribution? if you are creating the latter, how did you clone this to other devices, i'm assuming dd?

I'll probably prep the "gold master" SD card and then leave it to our SD duplicator.

I'm aiming to distribute SD cards that make some of our Pi Zero kits - Radio, Mood Light, etc - near frictionless to get up and running.

I found the use of Qemu + multistrap in this tooling easy to get my head around and I additionally wanted a stateless RAM disk so customers couldn't fubar it.

I didn't have much luck with Buildroot.

I evaluated TinyCore/PiCors too and I love the concept, but it's just a little contrived to ship to beginners. The way it mounts whole packages of overlays/dependencies is clever though- I am doing something similar (inspired by Open Pandora's pnd format) to add a custom build of VLC to PiratePython for our radio kit. Package the whole app + dependencies into a container image, drop it onto the SD card, mount and simlink everything into place.

I've had good luck with Alpine Linux:

https://wiki.alpinelinux.org/wiki/Raspberry_Pi

What about FreeBSD?

good support for raspberry pi or general arm support?

i found this, seems it's been supported for a long time. i'm not ecstatic about the wrappers in ruby/python for gpio, but then again i don't need it at this point in time.

https://wiki.freebsd.org/FreeBSD/arm/Raspberry%20Pi

I am a great fan of FreeBSD and I run it as my main OS on one of my two laptops that I use daily.

That being said, last time I tried to run FreeBSD on a Radpberry Pi was not a pleasant experience. That was several years ago though so things may have improved.

FreeBSD/arm64 now has official binary packages for 12-CURRENT. RPi3 runs in 64-bit mode with all 4 cores. For the most part, everything is good. But the vc4 KMS driver is not ported yet, so no GPU accelerated desktop for now.

I did a bare-bones buildroot[0] for Raspberry Pi to run tcpser[1], was a bit of hassle but not too unmanageable!

[0]: https://github.com/buildroot/buildroot/tree/master/board/ras...

[1]: https://github.com/FozzTexx/tcpser

wow, cool stuff, will check it out!

Why not use Arch Linux? The arm version is called ALarm (for obvious reasons).

https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-...

i was once a fan of arch linux, loved having the control of compiling everything. how well is this supported for raspberry pi? it's always nice to know that vendors have tested the exact hardware that i'm using extensively, even if it's just another arm device that looks and smells like the others. also, what package manager is used by arch? it's been like 10 years since i last used it.

pacman iirc

It still uses pacman.

I can't confirm this, but it feels like Arch linux has surged in popularity over the last ~3-4 years. I ditched Debian for Arch in 2016 and haven't looked back.

Maybe you are thinking of Gentoo? Arch’s core and community software are distributed as binaries that have been compiled for you.

The AUR provides additional software you can compile yourself but most of what makes the system run already built.

Wait... you're worried your OS will be so overkill you would create your own, but you're perfectly fine using Electron?

my requirement is to start up a browser, preferably chrome, and open up a certain page, this page is essentially like a digital scoreboard that is shown on a tv. electron embeds chromium and can start up an application super quickly, of course it requires nodejs and a bunch of nodejs libraries, but seems around 79M for the binary, my electron app requires a few nodejs libraries, but this is all text so nominal.

i explored using some scripts to autostart firefox and open up a certain page, but the fact that i needed kiosk plugins from third-parties just ended that for me, i don't want the browser frame to show up. electron is a bit overkill, but thus far it gets exactly what i want done. that piece will be optimized once i find the time.

I don't know if you have already tried this, but if not, try 'google-chrome --kiosk <page>'.

i also found that there are high level services that are needed to control the device, simple things like restart, changing ip's, upgrading, backing up, etc. i started to create a higher level library for these things using golang, basically a restful service, in this way it could be controlled via an application instead of having to use ssh, this is good for consumer like devices, but obviously opens up a can of security issues.

i will share the code (make it public via github) and write up my experiences developing for this device.

Did you look at Raspbian Lite? You could then install your choice of windowing system on top.

any suggestions for windowing system?

Void Linux might suit you: https://wiki.voidlinux.eu/Raspberry_Pi

I can't help but notice you've not tried standard Debian (raspbian isn't it), archlinuxarm nor, god forbid, any BSD.

I recently discovered dietpi and I am very happy about it.

http://dietpi.com/

lol from their home page:

> A tweakers wet dream!

Yikes

That is super cool.

When did 400mb linux become light weight?

Sorry, wasn't DSL debian based too... Have the basic things needed grown that much?

When it became less than 100% of the size of the smallest common OS for rpis.

Very nice! I've been considering attempting to port an OS that I've been writing to an ARM device like the Pi, but I've never really looked into it much. With that said, I'm surprised the booting portion is so easy, I was under the impression that for the Raspberry Pi (And ARM devices in general) it was more complex. I suppose though, this is started after the bootloader that's normally on the SD card correct? I would imagine that makes things a fair bit simpler if that's the case, since the bootloader would take care of all the secret sauce.

Do you have plans to implement paging eventually?

Yep the proprietary bootloader on the Raspberry Pi handles most initialisation leaving the SoC in a usable state. If you want to see how things like DRAM init work you can take a look at the open source reverse engineered firmware at https://github.com/christinaa/rpi-open-firmware . It isn't complete but it is just about enough to also boot the Linux kernel.

Thanks for linking that, that is one impressive piece of work!

I do plan to implement paging. I already manage all of the physical pages, I just haven't enabled the mmu yet as I wanted to get more fun features in first

I just want to thank you for posting this over here, I've been looking for something like this. I need something where I can develop and understand code as I go, so not a completed system, but that also provides final code so I can have something that actually works. I tried following the OSDev pages to try an x86 kernel, but I gave up when I couldn't get interrupts to fire.

This is fantastic. Especially the line-by-line explanations on separate pages. Thank you @jsandler18!

You can also leverage yocto/bitbake/openembedded to build your own linux distro for the pi

https://git.yoctoproject.org/cgit.cgi/meta-raspberrypi/

awesome.. might buy rpi 1 cuz of this

I think for the cost, a RasPi is always a nice board to tinker with. And having a bunch of MicroSD cards to experiments makes it easy to test multiple scenarios easily.

Reply on news.ycombinator.com