Gopher-OS: A Proof of Concept OS Written in Go | Hacker News Reader

Gopher-OS: A Proof of Concept OS Written in Go

github.com

172 pointsingve82 comments

Hi, I am the author of gopher-os. It started as a fun research project to learn more about the Go runtime internals and I didn't really expect it making it to HN.

If you take a look at the Go runtime sources you will notice that all the low-level arch/os-related bits have been split into separate files which usually invoke some syscalls (e.g. the memory allocator eventually calls mmap)

The idea I am currently investigating is to first provide an implementation for things such as physical memory allocation and virtual memory mapping which would ultimately allow me to bootstrap the Go allocator. From that point onwards the plan is to incrementally add more features and gradually initialize the rest of the runtime.

This is much more difficult than it sounds as all code must be written in such a way to prevent the compiler from calling the runtime allocator (no variables allowed to escape to the heap).

I started writing a kernel in Go a while ago (and since abandoned it due to not having time.. I got as far as having a simple built in shell that let you do ls and cat on a FAT filesystem). The part about memory allocation and virtual memory wasn't much more difficult than it would be if you were implementing your kernel in C or some other low level language.

The hard part is once you get memory and paging working, you'll need a syscall interface. If you want to be able to run Go userspace programs, that means you need to implement the syscall interfaces for an OS that's already supported, and can't just go out and design your own, and at that part it starts getting a lot less fun.

Could you instead write your own syscalls but also your own $GOOS target for the compiler?

That's possibly more work but it's the approach I would have taken as it seems more fun than just replicating Linux (or whatever) syscalls.

Sure, you could, but then you need to fork the Go compiler too (and hopefully eventually get your GOOS target mainlined, which seems unlikely unless your OS gets popular) or you could do your userspace in something other than Go and implement what you need for that.

My point was just the memory allocation and paging isn't significantly different than it would be writing an OS in any other language, it's once you get to the part where you need to implement other people's interfaces/standards to make a viable product that it starts bogging down (if you go that route. If you go your route, then you have two major projects instead of one: writing an OS, and adding a new OS target to an existing toolchain.)

I'm looking forward to seeing how well your dedicated OS supports the protocol. Are you following the RFC?

http://www.rfc-base.org/txt/rfc-1436.txt

Your project reminds me that I wish GOOS=efi to be a thing.

Ugh that would be completely amazing.

Wasn't there once a "tiny" runtime?

https://codereview.appspot.com/3996047/

How much of your codebase is error-checking?

https://anvaka.github.io/common-words/#?lang=go

(the top 4 by far are err, if, return, nil!)

Have you seen https://github.com/atmanos/atmanos ? It may be of interest to you.

Maybe I'm old, but I immediately assumed this was an OS built around the Gopher-protocol[1], whatever that would mean in practice.

Gopher is already an established name, so the naming seems a bit unfortunate.

[1] https://en.m.wikipedia.org/wiki/Gopher_(protocol)

The subsummation of the word 'Gopher' into the Go language eco-sphere irritates me also. I believe it has something to do with the Language having adopted the Gopher (animal) as their mascot?

Just because something is no longer in common/widespread use, does not mean you can come along and use its meaning or name on your shiny new product.

Gopher the protocol maybe niche these days, but it's not dead. The continued use of the term 'Gopher-XYZ' by the Go language people will only accelerate the death of the Gopher protocol, which although inevitable it does not mean it should be erased from history by something else.

There a tons of name conflicts for things that are more popular than the gopher protocol. If the creators wanted to avoid this they need to pick a unique name.

My pet peeve is Atom the protocol and Atom the text editor. They're obscuring a format and a protocol that's much more important for the web than another text editor is, and they're doing the world a disfavor as a result.

Even worse is when Justin Timberlake did that song "Cry Me A River", when there was already an excellent song by the same name. It just obscures the existing song, and does the world a disservice.

Pretty rude way to disagree with someone. I'd appreciate any criticism, but not sure what I'm supposed to do with this comment, other than feeling bad.

I'm getting the impression that he wasn't actually telling you to "cry him a river".

perhaps, have a laugh?

And I've been using Atom as my default username on local Linux installs since 2001. So now I invoke my user installed editor as

   /home/atom/opt/Atom/bin/atom

And to check it's running? NP! Just use `ps -eF|grep -i atom`

We need more letters

>which although inevitable it does not mean it should be erased from history by something else

Why do you think it's inevitable, if not exactly because it is to be erased by something else?

> Just because something is no longer in common/widespread use, does not mean you can come along and use its meaning or name on your shiny new product.

I get that each time a "ML" link is posted here. Sadly it's almost never about the various successors of Miranda.

There were many similar complaints when the Go language itself was announced, there already being a programming language named "Go!"

And also a 2500+ year old Chinese board game, generally referred to in English by its Japanese name (Go). Although the ambiguity of the name is beginning to popularize its Korean name, baduk (as in /r/baduk)

So was windows, but that didn't stop Microsoft. ;)

Or the 3 or 4 competing apps named "paper."

The worst part about the GOPHER protocol is that actual Gophers couldn't even use it, because they don't have opposable thumbs!

Yet another instance of the programmer establishment thinking they know what's best for everyone...

This is a common misconception. The GOPHER protocol (RFC 1436) does not actually require the use of opposable thumbs, it's just a common implementation detail to do so.

Even worse, there's already an animal called a Gopher! Oh the confusion... /s

Bikeshedding, we can also start discussing why `println` doesn't print anything on the printer, apple doesn't sell any fruits and their ipad is not really targeted to women during period, microsoft has no double glazed windows offers, oracle without cloud based prophecy api and why ml lang and machine learning are stealing established, widely deployed in most kitchens millilitres measure - nobody gives a crap.

nit:

println does indeed print if you're using a teletype. The fact that you're probably not using a teletype is simply a matter of preference on your part.

Out of sheer curiosity, what would it take to hook up a real teletype to my computer today? What physical port would it even use?

Teletypes used 20mA current loops

https://en.wikipedia.org/wiki/Digital_current_loop_interface

Teletype current loop ports were common on machines up to the 1970s, but by the 1980s had largely disappeared. The serial port card on the original IBM PC and the XT had 20mA current loop support, but it was very rarely used, so AT machines onward removed it. (MIDI is also a current loop, but MIDI uses 5mA rather than 20mA, plus the protocol is different.)

You should be able to connect a teletype to a modern computer by using two conversion devices, one USB-to-RS232 and the other RS232-to-20mA-current-loop. Then the teletype will appear to the computer as a (very dumb) serial terminal.

There was already a programming language called "Go" before Go, so I'd say this is actually the customary way.

I don't know anything about OS development, but how do you implement an OS in a language whose runtime depends on an OS?

Looks like it's a matter of not importing almost any libraries other than "unsafe".

Not only that, but you'd have to take care not to do anything that would trigger an allocation, since you don't have memory management.

Just on the very lowest layer, then GC is just yet another kernel service.

There are lots of GC enabled OSes and research papers to learn from.

On the lowest levels only. The rest of the OS can be GC'd. See Oberon, JX OS, and House in Haskell for instance. That Go has a low-latency GC makes it even better for that usage.

http://programatica.cs.pdx.edu/House/

Well, the Go runtime is written in Go itself. And you can write Go without the runtime, the same way the compiler is.

The hard part is that the runtime is indeed build on OS primitives (syscall mostly), that you have to implement yourself.

You have to forgo all the niceties of having a runtime but you can do it. Definitely not the most productive use of Go, but it's fun and you learn a lot.

> And you can write Go without the runtime, the same way the compiler is.

The compiler does depend on the runtime AFAIK

> The hard part is that the runtime is indeed build on OS primitives (syscall mostly), that you have to implement yourself.

This is what I was referring to. I don't know how you would do this without patching the language.

By using a mix of cross-compiling and some Assembly help, just like C needs some help beyond what ANSI C specifies.

> Well, the Go runtime is written in Go itself.

That's a bit of a stretch. The runtime uses special pragmas that are not available to normal programs. I don't think it would be possible to write Go's GC in Go unless these special pragmas existed:

https://github.com/golang/go/blob/57df2f802f0417f08100ff8002...

Also, there are magical variables in the runtime that the compiler knows about and special-cases: https://github.com/golang/go/blob/57df2f802f0417f08100ff8002...

> And you can write Go without the runtime, the same way the compiler is.

How would you write Go without using the GC?

That is the magic of cross compilation.

There is already a compiler that knows those special pragmas, one just needs to add a new bare-metal backed to it.

As for writing Go without a GC, just like in any other GC enabled systems programming language, by not using language features that require GC in the lowest layer, which the other packages then depend on.

Example in Oberon, https://people.inf.ethz.ch/wirth/ProjectOberon/Sources/Kerne...

Also even ANSI C requires some Assembly or compiler extensions to fully implement libc.

The entire runtime does not depend on the OS. C's run-time is very minimal for exaple. You have a stack memory, heap etc that you setup in protected mode. Your OS will manage the page-tables, process creation etc.

"Go" binary will basically compile into machine code which can be run on bare-metal.

I am doing this currently: https://pdos.csail.mit.edu/6.828/2012/schedule.html

And it's been eye opening to how everything works under the hood!

Type safety.

I advise you to read Project Oberon from Niklaus Wirth, originally published in 1992, revised in 2013 for targeting a FPGA instead of the Ceres workstation that was used at Zurich Institute of Technology (ETHZ) during the 90's.

https://people.inf.ethz.ch/wirth/ProjectOberon/index.html

http://www.ocp.inf.ethz.ch/wiki/Documentation/Front

A complete graphical workstation OS used by the IT department for their OS programming classes, language design and even by the non technical personal at the department.

What does this have to do with the runtime of a specific language depending on an OS?

All runtimes can run bare metal, it is just a matter how much Assembly they might require for the HAL.

Runtimes are a portable OS, that just by convince and practicability reasons happen to run on top of an existing OS.

That is the whole premise of Unikernels, to go back to the days when the language runtime was the OS, like on the Xerox environments.

You generally don't. Thus the "proof of concept".

Though to be fair a lot of the standard C libraries have there own "Linux" version in the Kernel so the OS can do things without using the OS...

http://lxr.linux.no/linux+v4.10.1/include/linux/

Probably should have let this develop a little further before posting it.

No way, raw stuff on HN, moar please. Lots of learnding for all

Is Go suitable for OS/Systems development? Inline Assembly would be nice to have.

I'm guessing a project like this will need to implement the Go runtime given that (I'm assuming) it uses posix (Or windows etc.) system calls.

Go has an assembler, but its goal is not to build, say, a bootloader:

https://golang.org/doc/asm

The Go assembly language is actually one of the things that make doing something like this difficult.

Go uses the Plan9 assembly syntax but doesn't support GNU asm syntax. gccgo uses GNU asm syntax but doesn't support Plan9 assembly.

The Go linker doesn't allow you to not link in the Go standard library runtime, which means you need to use gccgo to write a kernel.

The end result is that you can't do something that's buildable with the standard Go toolchain, and even if Go were to add a "don't link the runtime" option there'd be no way to incrementally port your ASM over one bit at a time. It's all or nothing.

Just like C in the 80's and 90's, and it wasn't a problem for its adoption.

I take a slightly different approach for my implementation that allows me to use the standard go toolchain. I use nasm for the early ASM code and patch the go build tool's output (see: https://github.com/achilleasa/gopher-os/blob/master/Makefile...) to bypass the link step and replace it with a manual call to ld that links the go object files with the output from nasm.

I will be talking about this approach in more detail in GolangUK '17.

> I will be talking about this approach in more detail in GolangUK '17.

Sounds great. Any chance for you to come to dotgo.eu?

Do you know if your talk going to be online afterwards?

Yes, the talk will be recorded and posted online with the slides and some getting started code.

Interesting project.

What's all this voodoo inside Makefile, I would love more comments in there. I think that would be the first step for anyone wanting to contribute, to understand the Makefile.

    @echo "[go] compiling go sources into a standalone .o file"
    @GOARCH=$(GOARCH) GOOS=$(GOOS) go build -n 2>&1 | sed \
	   -e "1s|^|set -e\n|" \
	   -e "1s|^|export GOOS=$(GOOS)\n|" \
	   -e "1s|^|export GOARCH=$(GOARCH)\n|" \
	   -e "1s|^|WORK='$(BUILD_ABS_DIR)'\n|" \
	   -e "1s|^|alias pack='go tool pack'\n|" \
	   -e "/^mv/d" \
	   -e "s|-extld|-tmpdir='$(BUILD_ABS_DIR)' -linkmode=external -extldflags='-nostartfiles -nodefaultlibs -nostdlib -r' -extld|g" \
	    | sh 2>&1 | sed -e "s/^/  | /g"

edit I think I understand what's happening here, funny how writing it down helps one understand. It's just building up a build script with sed and executing it. The thing was I didn't know sed could do that. TIL

This seems to be in very early development with the first commits in late March and a single line readme. Interested to see where it goes and what problems they encounter.

nice! btw, wrt the `Memset` function, if you have something like

    func foo(b []byte) {
        for i := range b {
            b[i] = 0
        }
    }

the compiler turns it into a `memclr` call which can be pretty fast: https://github.com/golang/go/commit/f03c9202c43e0abb13066985...

Perhaps there could be a performance boost to a simple for-range loop if `value == 0`?

Good luck with the project, nice to learn about it.

A lightweight Go OS would be nice for virtualization environments. Deploy your Go server/service with the minimal needed OS into the cloud...

Perhaps it can have a JS shell?

And store data in HTML elements instead of files?

With the S not capitalized in "Gopher-Os" I read it like a cereal (like Cheerios).

I've never really understood OS:es. Why are so many people creating them, yet so few have any success? I've must have seen a dozen OS:es over the past two years on HN, most developed by people having no confidence in calling it a serious effort.

Why care about OS:es when there is no room for anyone on that level of abstraction apart from a handful developers at MS, Apple and Google?

Most of the ones I've seen were learning experiments. A really really good way to understand something is to build it.

If you think of an OS as solely something that sits on your desktop, then sure, but this is fairly myopic (and defeatist). There are lots of small, proprietary and often embedded operating systems that rely on the same principles established by present and past OS.

Writing operating systems is also a great way/introduction to work on operating systems, and it's a fairly distinct area of development. It's hard to get day-to-day overlap if you're working on things that don't have the constraints and operating system does.

It's usually a learning project. The same way tons of people implement web servers but ultimately use something out of the box in production.

https://github.com/klange/toaruos << example of a hobby OS taken farther than most I've seen.

ever heard of fun? or learning? not everything needs to have the potential for monetization to be worthwhile; it can just be interesting. this is actually something that has been difficult for me to internalize, but i am much happier for it. no less ambitious, but i've found that there are a lot of nice (pointless) things that contribute to my happiness in addition to success.

Operating systems' success is fundamentally based on their popularity. Popularity resembles an exponential distribution, therefore there can only be a small number of popular ones with a long long tail of less and less popular ones. Here, popularity is a positive feedback loop, where the more popular it is the more developers target it, and the more it's targeted the more it's developed internally, which makes it better and therefore more popular.

That doesn't mean it's useless to develop them. If you set your sights low (playing around, learning, academic) you can still benefit from them without them being a huge success by popularity standards.

Having no confidence in the project is part of the path to success

"I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu)"

Hey, good luck with it! I'm thinking maybe I'm just wired differently. If I have a side project or hobby I still want to feel maybe not success but a sense of completion.

An OS project I suspect will always end when you no longer have time for it or are stuck somewhere. I'd be annoyed by that.

@skocznymroczny was actually quoting a post from Linus Torvalds when he first released Linux.

It might have been helpful for GP to note that the quote comes from Linus Torvalds. He said it while he was first developing Linux

Your reply, while I'm sure it is genuine, couldn't be more on-point by being more wrong.

You really don't know, and cannot know, where the next Linux is going to come from. I don't know if this will be it, just by considering probabilities, it probably won't be, but it could.

Even at the worst case, the author of a project like this is going to learn A LOT from it, and the people who read through the source code are also going to learn a lot. Writing an OS in a higher-level language absolutely is beneficial to people who want to learn more about how hardware works and interacts with the OS, without needing to delve into something as complex as the Linux/BSD kernels.

At a minimum, I think projects like this directly create contributions to other "real" kernels by teaching developers how to do kernel/OS programming.

> Why care about OS:es when there is no room for anyone on that level of abstraction apart from a handful developers at MS, Apple and Google?

There is as much room as there are people frustrated with the "state of the art."

Edit: Have you seen this apple commercial https://www.youtube.com/watch?v=OYecfV3ubP8. Yes it is from 1984 and you might think it doesn't apply.

The reasons aren't so different from the reasons for writing any other software.

1. Even if the project never replaces the other software you use, you gain a better understanding of that other software by solving some of the same problems, making the same mistakes, learning the same lessons.

2. You may have special requirements that don't make sense to add to an existing project aimed at a broader audience. This is where a lot of niche microcontroller OSes come from.

3. You have a crazy idea that would be hard to explore within the architecture of an existing project. If it turns out to be crazy enough to work, then maybe people will do the work to migrate it to the mainstream.

4. Everyone has a little dream that the masses will find great value in their work and flock to it, supplanting the status quo. Like becoming a billionaire, it essentially never happens, but you can't blame people for trying.

5. fun.

A lot of people -myself included- enjoy solving puzzles for fun. So I often set myself weird or obscure challenges to solve with code.

Why care for computers when there's a worldwide market for maybe five, tops?

That's not a good comparison. The original quote was made in reference to the total number of machines, not to the number of models.

Lookup Redox

Reply on news.ycombinator.com