Introducing Kirk | Hacker News Reader

Introducing Kirk

ian.roon.io

41 pointsIanChiles46 comments

I take quite a big issue with the statement "The lack of information and documentation on developing OSes is, in my opinion, a pretty big problem"

There are entire forums for this, ie osdev http://wiki.osdev.org/Expanded_Main_Page

There are entire books for this http://www.amazon.com/s/ref=sr_ex_n_1?rh=n%3A283155%2Cn%3A5%... amazon lists about 57,000 results for operating systems books

You will get out of LFS what you put into it. It tells you how to download the source, extract and install it. When you have the source, nothing is preventing you from looking at it.

I also take issue with "how can we hope to continue developing and innovating on the OS level when it’s practically impossible to figure out what anything does or how it works."

There are general concepts that you will learn from getting a book on os development (including the kernel) and writing a minimal kernel yourself will give you insights into the work that goes into a full kernel such as the linux one. The concepts, not the specifics are what you need to learn, and you do this through theory, you pick up the rest from implementation (but starting with a kernel, half of the os is already developed fof you).

Absolutely agree with you. I struggled with wording this for the longest time - and when I talk about OS development, I'm referring much more to the userland development - package managers, core utilities, shells, init systems, and the like, than I'm referring to kernels. Sorry for the confusion.

I think some of the less explored portions of LFS contain well-documented reasoning for implementing some of these things yourself. For example there is a single solution (out of the many ways to solve this problem) to the package manager dilemma in the hints section of LFS

http://www.linuxfromscratch.org/hints/downloads/files/packag...

which explains reasoning for the package manager and how to implement it yourself (in a much less copy-compile-execute manner than LFS itself).

Implementation is a great way to learn but generally people will survey the literature first to understand why things are implemented the way that they are. Understanding what all package managers have in common will allow you to distill it down to the simplest factor and implement that.

A lot of the core utilities and why they are expected should be covered in the standard (POSIX in this case).

I guess what I am trying to say is that implementation is a useful way to learn but it requires you to know the details whereas understanding the literature first will give you a better idea of what to implement.

Good luck with your project.

  For as long as I can remember, I’ve wanted to create my
  very own operating system, one which I have complete and 
  total control over. Because I have neither the time, nor 
  the skill to write my own kernel, this means that I’d 
  eventually land up creating a OS based on the Linux 
  Kernel.

Non sequitur. If you want complete and total control, you're going to have to learn the ins and outs of a kernel.

I applaud your decision to go forth with this, but is it really worthy of an HN self-post given that Kirk does not exist and is just a blog post at this point?

For me, at this moment, it's simply not a feasible goal to learn the ins and outs of a kernel in any reasonable amount of time. And Kirk doesn't just exist as a blog post, I've done a relatively large amount of work on it, but am just now starting to write about it by request of a friend.

If I were going to go back and learn the ins and outs of an OS again, I'd still go back and start learning from the bootloader to the kernel and then progress from there. The kernel is the OS for all intents and purposes. It sounds more like you want to learn how to create your own Linux distribution.

Don't fret, we've all been there and done this ourselves :). Some people catch the bug and create stuff like Gentoo and Arch.

I think it's a worthwhile endeavour to stick a distro together, however I agree with a lot of the other commenters here. If you really, truly want to learn about how an OS works, you're going to need to get your hands dirty and learn about the kernel. There are so many tradeoffs made on the kernel side of things which impact userspace, and knowing about them impacts the decisions you're going to make in terms how how you configure software and which software you're going to use.

If you can afford it and have a week to spare, I'd really recommend taking the Linux Foundation's one week crash course on the kernel. It talks about how the scheduler works, gives you insight into modules, how processes and threads work and hooks the kernel libraries give you for semaphores, mutexes, etc. It's expensive, but it's well worth the money.

http://training.linuxfoundation.org/linux-courses/developmen...

I'd actually really love to take a course like this, but a non-profit foundation charging $2750 for a virtual 4-day course?

I understand that they have probably have access to incredible kernel expertise, and hopefully the teaching is excellent, but even that doesn't warrant nearly $3000 for a 4-day course.

I mean, even training companies in the pharmaceutical industry doesn't typically charge quite that much for short courses! They're probably close, but is that really who the Linux Foundations wants to model their pricing on?

It's great if they can find corporations that can pay these kinds of prices, but it means that independent open source developers are virtually shut out, which means that many of the very people who would be interested in making contributions to the kernel will not be able to take what is probably the best introduction around to kernel programming.

Does anyone know of any good alternative resources for learning the kernel? There's a lot of good stuff on kernelnewbies.org (when it's not offline). Other ideas?

Yeah, I know it's expensive. I took the "in person" version of this course though and it was phenomenal. I'm not sure if it cost the same (the company I worked for payed for it), but if it is, it works out to $70/hr which is cheaper than getting specialized instructors in other disciplines like flight instruction.

For a virtual version of the course $2750 does sound pretty steep. It'd be nice if there were a "Khan Academy" type version (ie. pre-recorded) for a lot less.

Kirk does not exist, this is a blog post. Just a reminder to everyone like me who went looking for it.

And the blog post is about the same dust that was relevant 10 years ago and sparked the creation of LFS. Nothing new to see here.

> an OS without limits, built to be open from the ground up - with as open of a development process as possible.

I see the benefit of documenting each step. The result will be a pretty great tutorial, or at least a very interesting series of blog posts. But what is closed about the development of the current distros?

I hope that Ian is hoping to build a very different kind of distro, breaking the mold. NEXTSTEP was fundamentally a Unix, but everything about it was synthesized to create a totally new organism. Same with OS X. This kind of experimentation hasn't been attempted very much with Linux distros, which mostly amount to different installers and package managers. Etoile comes to mind, and apparently CoreOS is discarding much of the mold, but a completely new, holistically built system would be quite interesting to see. I suspect it hasn't been attempted much because the sheer amount of work is overwhelming. Writing a new shell is a lot of work. Writing a different init, shell, filesystem layout, and window manager, all unified around some new concepts, would be a lot of work. If that's what he's aiming to do, the result will be really interesting to use and learn from.

If there's any confusion, that's absolutely what I plan on doing - but I firmly believe that if it's kept simple/lightweight that it's more than doable. The GUI portions slightly worry me, but they're not happening until much later down the line

> Same with OS X

OS X is BSD with a fresh coat of paint. It really depends if you want to consider the whole experience (which I consider less stellar than most people would care to admit) or the internals (which are the same old unix stuff).

It's not a BSD kernel. It has the BSD userland, though.

Mach and IOKit and and a gazillion other things are hardly "the same old Unix stuff", if you're talking about the kernel. If you're talking about everything above that level, read Amit Singh's book if you really think there are just splashes of superficial changes at and above the POSIX layer. If you have and you still feel this way, I don't know what to tell you. I'm not saying it's all great, but there's a LOT there.

I like the concept, but wish this announcement would have come with at least one or two concrete steps taken towards the stated goal. Maybe your experience getting a basic shell working or something similar. At this point I have no reason to believe anything will come of this. Was maybe a bit too early for an "announcement".

Dreams: "I’ve wanted to create my very own operating system, one which I have complete and total control over."

Reality: Another Debian clone out there.

I assure you that I won't do that. If I wanted to make a Debian clone I wouldn't be doing this

> the development process of Kirk be accessible enough for anyone interested to jump right in and contribute

Would be a good idea to host it on Github.

Absolutely 100% will be, don't worry

I want to echo the recommendations of others: learning about kernels and writing your own would be a lot more valuable.

I'm speaking from experience when I say that trying to provide a userland sucks. You've said elsewhere on this page that you want to write your own shell and such; the problem is that people don't want IanSH, they want bash or zsh or whatever. If you write IanMACS, they'll ask how to run emacs. And I hope the libraries you write can compile Firefox.

Or if you just want to write init replacements, modify how configuration is managed... remember that Ubuntu (and Gentoo, but nobody cares about Gentoo anymore) did a ton of this and everyone I've talked to hates it.

So here's my concerns:

- If you intend to write an entirely new userland, you're going to spend a lot of time replicating existing work, only to have people complain about it.

- If you intend to do a new init, improve configuration management, change how logging works, whatever, your life will be consumed with modifying every program to use your stuff.

If you want to learn about operating systems, I'll echo the others and say write your own kernel and a minimal userland.

The open source desktop is kept alive mostly through donated spare time. This time is limited. We would do well to maximize the impact of time spent creating and maintaining open source desktop software.

Right now the standard language to build Linux software is C, which is great for performance, but not great for development speed. There are ports of things like GTK+ to other languages, but ports are less supported and usually don't take full advantage of their language's unique flavor.

To maximize impact per unit of time spent developing open source software, and also perhaps to entice more people into contributing to open source software, I believe it would be a good idea to build a desktop environment/widget toolkit with an opinionated API in Ruby. Anyone who has experience writing Ruby on Rails code with an automated red-green-refactor cycle likely could testify that it is pleasant and fast. Why not bring that sort of experience to building desktop software?

I considered Ruby and have worked heavily with it in the past, but it's simply too slow/heavy for this type of programming.

I can't believe I left this out of the post, but all/most new code will be written in Go - I'll explain why more clearly in my next post

If you're interested in how a simple version of unix works, I highly recommend checking out Xv6, a teaching OS based on UNIX version 6:

https://pdos.csail.mit.edu/6.828/2012/xv6/book-rev7.pdf

I applaud his efforts - but how is it so many people find main stream Linux distros that non-configurable that they need to create their own?

I mean, even distros like Ubuntu are highly configurable. You can kill off as many packages as you want and even compile your own kernel if you really want to.

Thanks! There's no problem with them not being configurable enough, the problem is with understanding how they work behind the scenes. The best way, IMHO, to address this, is by starting from nothing, and building it from the ground up - not just compiling things, but writing basic replacements for things like bash, init, etc. Along the way, this gives me room to try new ideas out on a much lower level - rather than just trying to hack them into Ubuntu/another distro.

What are your thoughts on Arch?

Arch is my distro of choice at the moment - but even it simply teaches how to copy/paste to make things work, never explaining what is happening when you run a command, or how it manages to do that.

How about Slackware? Personally I find it unjust how rarely Slack is mentioned nowadays compared to distros like Arch and Gentoo. From the conservative development, the BSD-style init system and minimalistic package management, I've always found it to be the most UNIX-like distro of all, and my personal choice.

In that case wouldn't you be better off with something like Minix that's designed for this?

Well, I'm much more interested in the state of the Userland, not the Kernel itself. And since Linux is the de-facto FOSS Kernel, it makes sense to base it around that, instead of trying to port the various compilers/tools that I've used to Minix. It's just easier to do on Linux, but I've tried using the FreeBSD Kernel, as well as XNU - both ended pretty horribly, so I'm sticking with Linux for the time being.

Minix comes with its own unix-compatible userland. It won't be as performant or featureful as the GNU versions, because it's designed for teaching - but I would've thought that would be what you'd want.

Why not try Linux From Scratch?

From the article: "LFS, despite how amazing it is, teaches very little about the inner workings of Linux and emphasizes copying and pasting to compile a Linux distro."

Honestly it is probably because there is some annoyance or another which comes down to the philosophy of whichever distro you are using.

Here are my slight annoyances:

ArchLinux: got more opinionated see /usr/bin and python3 in recent history, breaks a lot for a distro

Linux Mint: Broke firefox forced google custom search then DDG

Ubuntu: Unity Mir

Debian: Slow moving, long freezes in packages

Fedora: I don't need SELinux on my desktop box, font rendering (don't like infinality)

Opensuse: Need to use OBS to get packages that probably should be in a main repo

That being said I've never been that annoyed with any of the Distributions to actually go and build one.

I used arch linux for many years but after python3 broke everything I left for ubuntu 10.04 then after that went up I tried unity but didn't really like that, but I stuck with it until ubuntugnome was an official thing.

Right now Debian Wheezy on desktop, Ubuntu Gnome laptop, and both seem to work well enough. I've lost my want to customize my OS so much anymore since I'm not on arch linux and it becomes a little harder when you don't have something as easy to use as the arch build system.

Every distro I've tried has something that annoys me about it too. And this is why I keep coming back to Slackware: it's the distro that annoys me the least. It's also Linux in its rawest and least dilute form, a natural step up from LFS: you can edit config files and use standard Unix commands confident in the knowledge that the package system and distro tools won't clobber anything you don't expressly tell them to.

Whats wrong with FreeBSD?

Laptops.

I think it's a natural part of the learning process. Either abstract away (usually with leaky abstractions) or simplify by rewriting (usually with the same result once the complexity of the domain is factored in). In some cases it produces something better but usually not on the first iteration.

I completly understand the feeling and it's something that I wanted to do for a long time. I thought of starting with LFS but never got much time to play with it.

I'll definetly follow your posts about this.

I don't get why so many negative comments here. There were many times that I really wanted to understand and configure all my machine just for the sake of it.

Understanding and learning, it's simple. That's the main reason I use Arch, but even with it I don't usually go really under it to understand what program calls what and what is the function of program foobar, why it really is needed or how many switches and power can it give.

This is great :).

I'm curious about the result, I share his idea about LFS being a little bit too 'automatic' and the feeling of blindness that resulted. The joy of booting a system build from source was there, but didn't last long, in fact I quickly hit very weird network issues such as elinks seeing the network but not wget, reinforcing the notion that I there was a need for a different kind of information about the system.

I don't understand. You want to do a non-GNU/Linux OS or what? Or "just another GNU/Linux distro"?

It sounds more like Kirk/Linux.

Which is exactly what it'll be. I really hope to keep the userland portable - so Kirk/FreeBSD or Kirk/Darwin could happen in the distant future

Arch Linux strikes perfect balance between usability and learning. I am looking forward to what you have to say.

How in the hell is this kind of vaporware garbage on the front page?

Shut up and code.

Reply on news.ycombinator.com