Serenity OS: Interview
corecursive.com
corecursive.com
But also big props to Andreas for speaking so candidly about the back story of Serenity. I feel like there are certain things we don't talk about in tech and that can change.
I love the idea that programming, esp side projects, can be a therapeutic thing.
I'm definitely a fan of talking more about things that we've been hiding from each other.
On that topic, one of the most common questions I get personally is "which drugs did you use in the past?" and I recently made a video[1] answering that question. It was a difficult thing to get through, but the overwhelmingly supportive response has been amazing.
For anyone who's struggling and is afraid to open up about it, know that there's a community of millions around the world who will welcome you with open arms.
Now more technical questions ... for my own sake of understanding and learning.
What would you say that were most important aspects of re-implementing (subset of?) C++ runtime that must be appropriate for in-kernel usage?
Where would I look in the source code to learn about SerenityOS paging algorithms, and virtual memory management in general?
Something that I messed up originally and we're now correcting incrementally is the handling of allocation failures. There are many places in a kernel where errors don't have a natural propagation path (IRQ handling for example), so operations that may fail (like allocation) must be avoided.
In general, heavy use of C++ features like templates and RAII has been very helpful for making kernel development productive.
> Where would I look in the source code to learn about SerenityOS paging algorithms, and virtual memory management in general?
You'll find the kernel memory management code in https://github.com/SerenityOS/serenity/tree/master/Kernel/Me...
Quick breakdown:
- Every Process has an AddressSpace.
- An AddressSpace has a number of Region objects, each with a virtual base address, size, permission bits, etc.
- Every Region has an underlying VMObject.
- VMObject is virtual and can be AnonymousVMObject (MAP_ANONYMOUS) or InodeVMObject (MAP_FILE).
- Cross-process memory sharing occurs when two or more Regions in separate AddressSpaces use the same underlying VMObject.
- MemoryManager handles physical page allocation, fault handling, page tables, etc.
> Something that I messed up originally and we're now correcting incrementally is the handling of allocation failures.
What in particular? Can you use exceptions reliably in the kernel land? I presume not unless you pre-allocate some hard-to-exhaust space for them.
Have you maybe thought of using LSBs to encode an allocation failure within the pointer itself? It's an entertaining idea I had for some time now. In 64-bit byte-addressable architectures one can take advantage of suitable alignments and have 3 least significant bits used to encode 8 different states. In a similar vein one could also take advantage of MSBs given the virtual address memory space limit of nowadays CPUs.
I haven't thought about encoding allocation failure as a low pointer value. I suppose you could, but it'd only work for pointers with sufficient alignment. Character/byte pointers often start at unaligned offsets when not coming directly from an allocator. Either way, I'm not sure compressing this information would solve any real problem we're having.
[1] https://github.com/SerenityOS/serenity/blob/master/AK/Error....
[2] https://github.com/SerenityOS/serenity/blob/master/AK/Try.h
> Character/byte pointers often start at unaligned offsets
Yes, but the address of that pointer is still 8 bytes long and idea of using 3 LSB of that address to encode the state is still applicable, isn't it?
> How's that possible? Isn't that an UB?
"Unaligned" was not the best word to use. What I really meant was byte-aligned:
char* s = strdup("Hello world!");
char* p = &s[1];
Yeah, I figured just few moments after I replied what you actually meant. That's why I edited the post above.
Well, to make it more robust one would have to wrap it in it's own pointer type but I think it's doable. I don't know whether or not it is a wise choice to make it a dependency in the OS though :)
Really glad you guys got together for this.
It was a great episode to make I'm always looking for devs with interesting backstories to share so if have any ideas shoot me a message.
One of the things I look forward to at the start of every month is SerenityOS's monthly update, this was the latest January 2022 update where they've done a port of Half Life and their continual improvements to their libGL implementation: https://www.youtube.com/watch?v=DL-RCV5a-C4
Due to the sheer gravity of the task it may not ever become a mainstream OS but because its entire source code is built from scratch and all maintained within the same repo, it's easily able to prototype new OS features you're unlikely to see anywhere else. In that regard it's possibly the easiest codebase to prototype new OS features given you can make changes to any part of the OS & view changes immediately.
What makes Serenity OS and others like it useful for this purpose is that they have much less unnecessary complication, so that much more of the effort can go into actually experimenting with the feature, as opposed to trying to slot it in compatibly with all the historical baggage.
Another useful quality is how they can demonstrate that a language some insist is unsuitable for kernels is, in fact, much better suited. You cannot write a whole competitive desktop OS almost single-handed without some decisive advantages. Serenity OS is coded in modern C++, and leans hard on its strong support for writing cleaner code.
An OS kernel, besides providing a platform, is a big program. Any big program makes heavy demands on programmer and language, independent of its purpose.
Isn't this a bit like what FreeBSD and other similar OSes do? To have a tightly integrated/tested base system with a set of software that it has, while maybe optionally supporting third party ports.
Now, in practice you almost always end up with binary distribution of packages rather than building everything from source due to time/resource constraints, but there is definitely an argument to be made about less fragmentation, like much of the GNU/Linux software has.
The Diablo port is a good example of this, whether it's missing C++ APIs, unsupported lang features, missing SDL port impl, toggling kernel features - it all gets done in a 1hr sit down session to produce a shiny new Diablo port.
https://www.youtube.com/watch?v=ZOzZ8R4gphE
Another nice quality of SerenityOS is that the entire OS is written in modern C++ with everything being written from scratch to use the standard AK libraries which the entire OS and components use, i.e. there isn't any existing App that brings in their own string types or other utils.
Andreas with Serenity not only sees the system through but can modify any part of the layer to make the abstraction of each layer just right.
I've been doing this with Flutter recently. Turns out it is both a toolkit and a package/platform. If you take out all the 'flutter' code, you're left with something that looks like sdl or glfw, just written in Dart. And it 100% has made me a better Flutter dev to do so. I now get a lot more of _how_ Flutter's drawing loop works, and how the framework beneath reduces all of app startup to one `runApp` call.
Not being able to inspect the insides of a tool easily is a major turn off to me on anything.
https://warosu.org/g/?task=search2&search_text=awesomekling&...
Definitely among the coolest things I've ever seen on 4chan.
It was a different time, but Linux did not come so far so fast.
Damn Scandinavians!
Yeah it's a state of mind, you do something for and with a community and not exclusively for your personal gains. In the US that was often compared to communism (just listen to the questions from reporters in the early day of Linux)
It's not a political thing, maybe more as a side effect of living in a society that promotes well-thougth actions and good planing and gives you the tools to achieve things.
That being said, I consider the the Scandinavian "socialism" the perfect combination of communism-light and capitalism-light: you can get rich if you work hard, but your boss can't get extremely rich by paying you far below a living wage. In the long run it's a win-win for everyone.
I know, but the US-Press always try to make something political, even when it's a cultural thing (and the only way to survive in the past..working together).
>That being said, I consider the the Scandinavian "socialism" the perfect combination of communism-light and capitalism-light
Absolutely on your side...and as someone from Switzerland i think we tend more and more towards the Scandinavian model.
> living in a society that promotes well-thougth actions and good planing and gives you the tools to achieve things.
Sounds beautiful to me.
That's just as common in the U.S. - but you need a strong community of practice with high standards of achievement to really achieve this mindset. (In the real world, this would of course be an actual, physical community bound to a well-defined place, ultimately with similarly strict norms of acceptance, behavior etc.) And that can get in the way of contemporary norms against perceived "gatekeeping". Though it's interesting to see that some communities of practice (e.g. the one which is forming around Rust and Redox OS) are being more successful in navigating this challenge.
No it's not....but it exists.
1. https://www.youtube.com/watch?v=Wr3xN52QYtA
What's the best way to look at the beginnings of the project?
There's a first git commit saying
"Import all this stuff into a single repo called Serenity."
And it is /completely/ from scratch - its own libc and libc++, own kernel, ui framework, browser (and I mean from scratch not Yet Another WebKit/Chromium wrapper), etc
There are now a few regular core contributors now so it's not just one man vs the world
SerenityOS demo at Handmade Seattle 2021 [video] - https://news.ycombinator.com/item?id=29270776 - Nov 2021 (180 comments)
SerenityOS: Year 3 in Review - https://news.ycombinator.com/item?id=28817599 - Oct 2021 (46 comments)
Not-a-Linux distro review: SerenityOS is a Unix-y love letter to the ‘90s - https://news.ycombinator.com/item?id=28219800 - Aug 2021 (113 comments)
SerenityOS: Graphical Unix-like operating system with classic 90s UI - https://news.ycombinator.com/item?id=28206840 - Aug 2021 (129 comments)
I quit my job to focus on SerenityOS full time - https://news.ycombinator.com/item?id=27317655 - May 2021 (249 comments)
SerenityOS: Writing a Full Chain Exploit - https://news.ycombinator.com/item?id=26115141 - Feb 2021 (9 comments)
SerenityOS: A love letter to '90s user interfaces with a Unix-like core - https://news.ycombinator.com/item?id=23911180 - July 2020 (1 comment)
SerenityOS Update (April 2020) - https://news.ycombinator.com/item?id=23037581 - April 2020 (1 comment)
Introduction to SerenityOS Programming - https://news.ycombinator.com/item?id=22479132 - March 2020 (43 comments)
Pledge() and Unveil() in SerenityOS - https://news.ycombinator.com/item?id=22116914 - Jan 2020 (28 comments)
CTF writeup: First published SerenityOS kernel exploit - https://news.ycombinator.com/item?id=21918351 - Dec 2019 (2 comments)
SerenityOS: From Zero to HTML in a Year - https://news.ycombinator.com/item?id=21212294 - Oct 2019 (52 comments)
Serenity OS update (August 2019) [video] - https://news.ycombinator.com/item?id=20851356 - Sept 2019 (2 comments)
SerenityOS – a graphical Unix-like OS for x86, with 90s aesthetics - https://news.ycombinator.com/item?id=19986126 - May 2019 (179 comments)
Serenity OS Demo (April 2019) - https://news.ycombinator.com/item?id=19813417 - May 2019 (1 comment)
Serenity: x86 Unix-like operating system for IBM PC-compatibles - https://news.ycombinator.com/item?id=19537807 - March 2019 (83 comments)