Serenity: x86 Unix-like operating system for IBM PC-compatibles | Hacker News Reader

Serenity: x86 Unix-like operating system for IBM PC-compatibles

github.com

335 pointsingve83 comments

Hey guys, I'm the author of this project. Surprising to see it on HN but life is surprising sometimes!

I actually made a little demo video about this system just yesterday if you'd like to see it running: https://www.youtube.com/watch?v=hE52D-zbX3g

This is some seriously intense stuff to build from scratch. I barely was able to get into x86 protected mode before I decided OS dev was too intense of a hobby for myself :)

I'd say congratulations but I'm not sure what to congratulate as there's numerous accomplishments here worth doing so for. Compositing windowing? Complete and functional network stack? Seriously, kudos.

Nice, easy to read code from what I've looked at, and impressive to have assembled it so quickly.

Were there any ideas you wanted to prove with your implementation?

Thanks! I didn't really have any big ideas starting out other than "I don't know what my computer is doing 99% of the time, what if I could drastically change that number?"

Just want to add a +1 this is incredible how much you've managed to do in such a short amount of time - a seriously impressive feat of engineering to even attempt, yet you've managed to come out with a working demo already.

This is just amazing, I'm very impressed with your OS. This also inspires me to begin creating things that at first thought might seem impossible.

How did you gain the knowledge to implement all these subsystems? Did you use any specific resources?

Just building a kernel and networking would have been awesome, but a GUI and all the applications is just mind-blowing cool.

Submissions like this is why I read HN. There are something intriguing about projects like this, that are small enough for you to have a chance of understanding how every part works.

>How did you gain the knowledge to implement all these subsystems? Did you use any specific resources?

I spent many years with a PC emulator (my other GH repo) being my only main hobby project. It's a reasonably correct 80386 with various peripherals and a BIOS. This project allowed me to take 80% of the skills acquired from the emulator, turn them inside out, and build an operating system from them. Of course, Serenity eventually outgrew the limitations of my emulator (no networking, no FPU, no VESA LFB...) and I've stopped trying to catch up for now.

Other than that, the OSDev.org wiki has been immensely helpful at times, and the vast number of open source operating systems you can look at for help/inspiration when stuck. :)

Two questions (pardon my ignorance on the topic, it's really exciting): - Can I write software for it? Is there a compiler or language included, or do I need to add it to the base code before compiling? - Would I be able to run it on bare metal someday (are any other drivers compatible?) or is this VM-only?

>Can I write software for it? Is there a compiler or language included, or do I need to add it to the base code before compiling?

At the moment you would need to add it to the base code before compiling. I've been working on porting GCC in increments. Last time I tried, GCC was able to build "Hello World" but somehow fell apart between assembly and linking for programs with more than a few functions. It's something I keep coming back to as it's quite draining to deal with the long build times of GNU tools for more than an evening.

>Would I be able to run it on bare metal someday (are any other drivers compatible?) or is this VM-only?

At the moment this is VM only. It could probably be made to run on bare metal relatively soon with a bit of hackery, but it's not something I've spent any time on since the VM environment is fantastic for rapid development.

Maybe another interpreted language could be an easier target?

- Ruby - Python3 or something else?

Does it support POSIX shell (yet?)?

- sh?

- Bash?

- zsh?

Bash was one of the first 3rd party apps I got running so it's pretty good. No job control though, and it sometimes gets confused about signals and TTY stuff. Haven't tried the others yet. It would be fun to get an interpreted language running.

Cool!

Thanks for the info. Maybe a simpler scripting or interpretive language (self-designed?) would be easier, rather than trying to bring in something complex at first. Something like a bash or BASIC that could make use of your GUI or events?

Do you know about TCC (Tiny C Compiler)? Might be easier to port that and build up from there.

https://bellard.org/tcc/

Can I share some observations? There are a few things I really love about this:

- You kept the end user in mind: You. The GUI looks like a mashup, in a good way: A Mac OS menu bar, Windows GUI feel and a NextSTEP launcher on top of a Unix like system. It shows that for every element, instead of simply recreating whatever system you decided to emulate, you took the parts you liked best from whatever system and simply recreated that. I haven't looked at the code too thoroughly, but I can probably find the same mindset there.

- You kept the scope tight. It's easy to lose sight of the bare minimum MVP, especially when working on a hobby project. The functionality showcased here tells me that whenever you started implementation of a feature, you built exactly what you needed, nothing less, nothing more.

- You also didn't get bogged down with premature optimizations. Yes, the PNG renderer is slow. But it does the bare minimum it's supposed to.

But most of all, this takes a crazy amount of time, skills and dedication. You've probably encountered quite some frustrating moments you had to chew through, but now you're here, presenting your work. And it's awesome!

Thank you for these observations, you've just distilled my personal programming philosophy into three concise points!

If you looked at the code, you would indeed find a similar "take what you like and leave the rest" style mashup. I spent many years working on WebKit, so you find a lot of WebKit-style patterns, containers and templates. I also spent some years working on Qt, so there's a huge bit of that in the GUI library. :)

But you rewrote it all from scratch in a “clean room” so as to not incur any license issues ;)

Truly amazing work. I'm curious if you know why the ping time to 127.0.0.1 varies so much between pings.

Probably because pinging 127.0.0.1 is actually cheating. I don't have a loopback network interface inside Serenity, instead pinging "127.0.0.1" will go through QEMU's SLiRP backend, ping lo0 on my host machine and then plumb the response back.

I didn't think of this when making the video, I simply typed "ping 127.0.0.1" out of some reflex. So basically, I need to make a loopback interface. Putting that on my TODO list, thanks for noticing this!

The README here is a model (or perhaps a superfluity) of modesty. This person has, apparently, written a complete OS from the metal up to a usable GUI, single-handed, AND done it in about 6 months -- at least, the oldest commit in the Kernel branch is October 2018. Well, I'm impressed, anyway.

Modesty in the comments too :)

https://github.com/awesomekling/serenity/blob/master/Kernel/...

/* * Really really really Q&D malloc() and free() implementations * just to get going. Don't ever let anyone see this shit. :^) */

Sorry, what is "Q&D?"

Quick and dirty

quick 'n' dirty

Oops, I forgot that comment in there huh!

I keep postponing writing a better malloc because every time I run into trouble with this thing, I spend 5 minutes optimizing away whatever the current problem is and buy myself another month of not having to write a better malloc. Someday though.

This might actually be the most impressive thing I‘ve seen here in a while.

It goes on the same list as the person who made their own video game console on breadboards: https://news.ycombinator.com/item?id=19393279

These are the most inspiring HN posts. They’re not marketable startup ideas, but the “lone hacker” working on something huge that I wouldn’t even consider, and then getting it to work, is very impressive.

I know there is currently in the air a zeitgeist against declaring it, but I still maintain that almost noteworthy humankind achievements happened like this, or in very small teams of highly aligned individuals.

Now of course, he is standing on the shoulders of giants, so to speak, but hasn't he become very close to a giant himself now?

It is almost always true that your daily behavior speaks your deepest beliefs, and to put something like this together as quickly as the OP did...well, awesome work OP maybe now you can work on flying cars :)

Thank you for your kind words. I think the world has plenty of boastful OS developers already. I'm just someone having a good time programming.

To clarify the repo history, the bulk of the codebase is from October 2018 forward, the one exception being a tiny bootloader+kernel I worked on for two weeks (and originally lost interest in) earlier last year and then imported into this repo.

This is excellent. I was expecting to see a screenshot of a command prompt and was pleasantly surprised to see GUI and applications.

"the goal here is a marriage between the aesthetic of late-1990s productivity software and the power-user accessibility of late-2000s *nix" That sounds right up my alley. Will give this a try!

I remember when home computing was full of neat hobby OSs. It was fun. Then somewhere around the last 20 years everyone stopped writing kernels and just built Linux distros instead (and these days you don’t even see that much variety in Linux any more either).

Anyhow, this project looks really inspiring. At least from a quick cursory glance. Looking forward to digging a little deeper tonight.

> (and these days you don’t even see that much variety in Linux any more either)

I beg to differ! While I do see that there are a a handful of very popular distros, there are tons more out there!

Spend a few minutes poking around DistroWatch[0] and see if you don't agree.

0. https://distrowatch.com/

I'm not on about quantity of distros, I'm on about variety. A thousand Ubuntu derivatives doesn't add much variation. systemd hasn't helped much here either because there used to be a lot of variety in how distro's would implement sysv. There even used to be distros that rolled their own file system hierarchy.

Some might see the grand unification a good thing for interoperability and developers - and I don't disagree with them. But it was fun looking at the different ways different distros would solve the same problem. Whereas now the biggest difference between a lot of distros is a great deal more superficial.

I agree that the majority of distros are based on debian/ubuntu/fedora but there are still some significantly different ones out there. The most interesting/useful of these for me lately has been void linux[0]. It is based around runit (instead of systemd) and has a very nice native package manager as well.

0. https://voidlinux.org/

I'm not saying there isn't still some distinct ones out there. I'm saying in there is less variety than there used to be.

variety is the key description I used. Sure there are few outliers still but the rest are just systemd + (apt|rpm) and there this really the same as what Linux was like 20 years ago where even Debian and Redhat were distinctly different from one another. Even Arch has come into the fold somewhat over the years.

I think a lot of it comes down to scratching an itch vs. being productive. I'd love to see something new gain traction, I just don't see it happening. Linux pretty much controls the services market (so much so that WSL and LCW are a thing). Windows and MacOS rule the roost on the desktop, though I think ChromeOS have carved a niche that I wouldn't mind seeing mature, and not become Android Desktop.

There's some cool experiments out there. Who knows anymore. One of the great things of everything (even on the desktop) being a web app, is it's really made many applications more portable, it's also slowed creativity and broader adoption. Right now, 90% of the apps and tools I use run under Windows, Mac and Linux, that's pretty huge. New OSes without at least enough support for electron/v8 and a few other bits will hold them back imho.

It's good and bad. I cannot speak for anyone else, to me these things are very cool. In the end, I have work to get done.

Having worked on the Windows kernel team, I can say this impressive stuff. It seems early enough to pick a specialty for this OS that it could do well that others may not.

The author has discussed his system for months in 4chan.

what board/generals?

The /g/ daily programming threads.

It appears to be extremely bare bone. For example the TCP socket is like UDP with hard coded IP addresses. It won't handle any packet reordering or lost packets.

I haven't checked other parts, but if they are like that it would be more a very elaborate demo than something actually usable.

Still impressive even for a demo.

Your observations are of course accurate: many parts of the system are indeed quite bare-bones. I've been trying to bring up a huge number of subsystems in parallel by building enough of the scaffolding and infrastructure in each area for a larger system to grow around.

Now for the rest of my life, I will always have something to hack on. :)

> Now for the rest of my life, I will always have something to hack on. :)

This is such an inspiring and humbling sentiment!

Thank you for all your hard work on this impressive project.

Totally agree. Hat off.

> it would be more a very elaborate demo than something actually usable.

Depending of the definition of "usable." It's surely usable for anybody who wants to learn the basic ideas.

It's written from the scratch, and just a small part of the whole system.

To get the idea how the networking stack were introduced to the famous systems see, for example, how may people were involved in creating the Windows networking stack with TCP/IP, at the moment the already had other kinds of networking, and as Berkeley sockets already existed for years:

"The idea for Windows Sockets was first proposed by JSB's Martin Hall at a Birds Of a Feather session at Fall Interop in San Jose in September 1991. Around 40 companies were formative members of the Windows Sockets group. Representatives from these companies gathered in Seattle in December 1991 to debate and resolve the basis of Windows Sockets. Considerable effort between then and January 1993, which involved a substantial amount of Internet-based email discussion, together with several meetings and 2 interoperability testing sessions (WinSockathons) led to version 1.1 of the Windows Sockets specification which is the definitive reference point for Windows Sockets today. WinSockathon II proved that Windows Sockets 1.1 is a viable standard for application/network implementation interoperability."

Back then it was just a protocol among many.

On 16 bit platforms, we needed to buy it as additional software package.

Still remember learning a tcpip stack call Wollongong something. It is the best.

I totally agree with the author to do the minimum otherwise just one part can delay all.

http://sunsite.uakom.sk/sunworldonline/swol-10-1995/swol-10-...

"Comparing TCP/IP products for PCs"

"PathWay Access 3.2 (The Wollongong Group)" ... "The PathWay product installed more easily than all the rest. We found it amusing that Wollongong managed to squeeze its software into five disks while others needed up to 15 disks for essentially the same software"

That is the right way to build a hobby project of this magnitude in my opinion.

Rapidly MVP as much as you can do you have something usable. Then go back and refine the warts as and when they become a nuisance.

How would you bring up an OS? Implementing the bare minimum for each subsystem until everything is working and you're happy with it, seems preferable to spending 6 months on TCP sockets, then having to rewrite when it doesn't fit in with anything else.

This seems pretty incredible as a learning resource just as it is. I can't quite believe how much is here and how concise it seems - has anyone gotten it to run?

I really hope the author or someone else writes sone high level documentation! Just knowing the design decisions would be great!

Thanks. The design is mostly "discovered" through iteration. I tried making a few diagrams at one point but in a week they were already outdated, so I've given up on trying to document until things slow down.

Also of note, it appears to be a mostly (if not entirely) C++ codebase.

That's too bad. Needs more HolyC! (Rust would also do, in a pinch.)

Very awesome. Love that it’s a new take on a modern OS - including a GUI. I didn’t think I’d see even an entrant into that sphere from a non-corporate entity again.

Respect! This is a very impressive achievement for a single person and requires a lot of knowledge about operating systems.

If, like other people said, this was done in merely 6 months by a single person, let me bring some skepticism into the thread here. From writing a bootloader up to a functional UI with various functional user programs is not a 6-months job, unless you take many shortcuts.

It appears to have been done full-time, and a lot of shortcuts have been taken. I would say the main achievement is maintaining determination for that period of time without doing so as part of employment or a course of learning.

I've seen it done (multiple times).

I worked for a set top box manufacturer where the lead engineer wrote an OS and a GUI system, from scratch, for our platform in less than a year. It had a bunch of syscall personalities. One for vxworks so we could port stuff from our existing platform and one for Linux so that people could write drivers for it based on the O'Reilly Linux device drivers book (which was current at the time). It also had a couple of others for odd things that we needed. It was incredibly well written. It only took me a couple of weeks to port it to PowerPC and the first week of that was just phaffing getting my cross compiler working properly.

In contrast, we had another couple of programmers trying to write a web browser for that platform. After a year they had something that could render primitive web pages but there was still a good chunk of the web that didn't work "quite right". After two years they were still in a similar position.

Operating systems are a dark art, and the person who wrote Serenity is very talented, but OSes are, at least, a tractable and well defined problem.

Serenity Now!

Insanity later

Are there any pre-built distributions I can just plug into QEMU?

Here's what I can offer right now: unpack this in serenity/Kernel/ and then run "./run" in the same directory. Or just start QEMU manually with qemu-system-i386 -kernel kernel -hda _fs_contents

https://drive.google.com/file/d/1954ki8wzHT6axoj9vbFKj8XxFK5...

sweet, thanks!

and it works! Boots in like .1 seconds. Very neat.

One strange thing, in QEMU, the mouse is all crazy and I can't control it in a usable way. Jumps around.

That's weird! Maybe it's some confusion between the PS/2 mouse and the PS/2 keyboard both using the same I/O port.. Does it help if you give it a little keyboard input before the mouse input? What OS are you running QEMU on?

I tried giving some keyboard input before using the mouse, no change. I am using Debian Stretch. If there is a more appropriate venue where you'd prefer continue this conversation, let me know.

This would be quite an accomplishment for one person in six months.

I am skeptical.

Weird title. Made me think this was for the original IBM PC and its clones but it seems to require at least a 686 (Pentium Pro). By that time the term IBM PC compatibles was already obsolete.

This person has made their own fully fledged OS. Discuss that instead of bikeshedding about terminology.

That would be nitpicking not bikeshedding :)

(They do have a point though; IBM PC compatible implies it would run on an IBM PC...)

It's a form of bikeshedding, in that it involves discussing the superficial details that are easy to form an opinion about.

Oh my God they're literally bike shedding bike shedding.

I'm obviously gunning for inclusion in webshit weekly :-)

I mean, the term bikeshedding was created to describe a specific phenomena where people in an organization tend to put disproportionate consideration into trivial design issues. Particularly, as an example, the color of a bikeshed.

Here someone is just pointing out a semantically incorrect or misleading statement. And we can say that's nitpicky, to a degree. It lacks some of the other nuance of the term 'bikeshedding,' though. If anything that was nitpicking also was bikeshedding, then what's the point of inventing such a term?

And yes, I'm being nitpicky about the term bikeshedding, and proud of it :p

(P.S.: As far as comparing the word nitpicking to bikeshedding, I think nitpicking already implies disproportionate attention to small details; hence the 'nit' part of it.)

a nitpeccadillo - "phenomenon".

Thanks, for some reason I was thinking phenomenon was the plural.

I don't think it's an unreasonable point to raise. I would however reply that, in a world where most people's primary computer is some form of SOC with peripherals soldered to it in god knows what way, the term "IBM PC-compatible" to refer to an x86 system with a BIOS and VGA graphics and the rest of it is still meaningful and useful.

The "IBM" part simply means the standard that IBM created, and doesn't have to imply that the software runs on an original IBM PC - rather, it implies that software that could run on an IBM PC also runs on such a machine. Hell, DOS still runs on bare metal on a modern machine.

This is hardly a fully fledged OS. It's just the beginnings of one.

The 686 part reminds me that I was able to get the latest version of Debian to boot on a Pentium II with minimal work. 686 and onward has had 20+ years of support.

FWIW I can think of at least two non-IBM-PC compatible x86 platforms (the NEC PC-98 series, and SGI's Visual workstations) manufactured past 1996.

Tangential curiosity: what was different about them that made them incompatible?

They didn't have an IBM-compatible BIOS and used a different system bus than what came to be called ISA. They also had different and incompatible video and sound hardware, among other things.

Psion's Series 3 was a non-PC x86 device still available in the late 90s: https://en.wikipedia.org/wiki/Psion_Series_3

Microsoft released a non-PC x86 device in 2001, the Xbox: https://en.wikipedia.org/wiki/Xbox_(console)

Reply on news.ycombinator.com