After 5 days, my OS doesn't crash when I press a key
jvns.ca
jvns.ca
To attain this state of zen-itude, just look up something that you'd like to do where documentation is hard to come by ("how to write your own shell from scratch" is a funny one if you can want to learn a LOT about UNIX) and start your own project.
Related: とあるOS is a nice hobby OS too: https://github.com/klange/toaruos
@jvns: SPOILERS AHEAD, DO NOT READ FURTHER IF YOU WANT TO GET TO THE HEART OF THIS PROBLEM YOURSELF
As for your problem with the keyboard interrupt firing only once, I think (after a quick glance over the code) that you do not signal "end of interrupt" (EOI) message to the 8259 PIC chip after you have received an interrupt. I faintly recall having this same issue in my ordeals with the PIC chip, if you don't signal EOI, the PIC won't fire the next keyboard interrupt.
Although I think I was working with 8253/8254 PIT (Programmable Interval Timer) chip at the time but I think the same applies to the keyboard interrupt too, it's being handled by the PIC too.
Here's the relevant part from my project: https://github.com/rikusalminen/danjeros/blob/master/src/arc... https://github.com/rikusalminen/danjeros/blob/master/src/arc...
I'm sorry to spoil your "fun" debugging it.
Dealing with any of this is a fun exercise in being close to the metal but unfortunately has no application in modern computers because these days the built-in IO-APIC/LAPIC has replaced the legacy PIC chip.
* it's not been submitted for a long time,
* discussion there is closed,
* it's really good,
* it's technical, and
* it's relevant.
I commented about why I've resubmitted it here:
I have never been as humbled by any sort of programming in my life. This was back in 2004 so the resources available online weren't nearly as good. I basically figured everything out by reading Intel's x86 documentation from start to finish. It took me a solid 3 weeks just to figure out how to start talking to the ethernet card. I spent days reading the same few pages in the manual over and over trying to understand how to get everything to fit together in assembly & C.
After 9 weeks I was able to address & initialize the ethernet card after the OS booted. I never was able to send or receive any data after a solid 50 hours of work.
Oh yeah, and I got a solid A for my work.
The truth is, computers have become exceptionally complex. That is one of the reasons I've been building a medium complexity standalone system (ARM Cortex M based) to give folks something that is somewhere between 8 bit Arduino type experiences and 64 bit IA-64 or even ARM Cortex A9 level complexity. I realized when I started finding ways to teach my kids about computation that I was very lucky to have things like PDP-11's, VAXen, 68000's, and DEC-10s to play with which did not present this huge wall of complexity that needed to be scaled to get to the fundamentals. My target is a self hosted 'DOS' style Monitor/OS for the Cortex M4 series. Complex enough to host its own development environment and tools, but simple enough that you can keep it all in your head at the same time.
(Note: I understand why, that doesn't make it less ironic.)
Doing OS things is hard. In x86 it's extra hard
x86 is one kludge on top of another. Like interrupt chaining
These issues remind me of making a Sound Blaster card work on DOS (I was making a program to play a simple file in Pascal/C)
It's very complicated, and several trials to make it work (and no Stackoverflow, Wikis, etc)
From OS perspective, it's also overcomplicated: switching from real to protected mode is an exercise in doing every action in a long chain properly and in the right order or the whole undertaking fails. Remapping IRQs into interrupts that do not mess with the CPU internal exceptions, weird legacy memory layout with BIOS, a slew of CPU modes (real/protected/PAE/long/etc), crazy format of entries in Interrupt Descriptor Table (where four bits of a pointer may be stored separately from the main part), where bits are often used differently for different contexts; redundant built-in features like poor multitasking support (Linux and other sane kernels try to avoid it as much as possible, but it's still is not possible to go into kernel mode from userspace without filling kernel stack pointer into a TSS entry), memory segments (in protected mode, where a pointer can address the whole virtual memory unlike real mode), four privilege rings (whereas all modern OSes use two: 0 for kernel space and 3 for userspace).
All that does not feel like it should be and often it only gets worse.
Very true, and while the X86 is patch upon patch upon patch, in the designers defense, the intent on any 286+ Intel CPU was simply to was to use "real mode" to provide a bootstrap CPU to use to setup enough of the protected mode data tables to be able to actually enter protected mode. The intent was that entering protected mode was a one way, boot time only, street. Could they have made it simpler, maybe, but they were also viewing it as a one time effort that once you got the switch implemented, you didn't need to touch that code again.
[edit for clarification]
On the x86 side I was surprised to find an unnecessarily cryptic assembly language, little-endian words and a byzantine memory model for starters.
Hacks upon layers of hacks and not the intuitive and aesthetically pleasing solutions seems to win out in most cases.
Agreed, 1000 times over. To the author: WELL DONE.
The jumps between real mode, v8086/protected mode, and long mode are absolutely mind boggling.
Here's the best part that I remember from implementing Intel VMX support in a VMM: once you've booted the VMM and are ready to do a VMXON to load a guest, you will realize you have a problem. Specifically, Intel realized a while ago that "real mode" sucks and is obviously not the "real mode" that actual applications expect to run in, so they introduced v8086 mode in addition to real mode.
The problem, of course, is that booting an OS still requires real mode, but you can't run a guest in VMX in real mode, only v8086 mode. But all those things that Intel "fixed" in v8086 mode? Yeah they will totally keep you from booting.
So you're chugging along in v8086 and your guest does something that it should absolutely be able to do and BOOM -- you've faulted. So what's the solution? Write your own emulator for a Real Real Mode(tm) in your host, capture the faults, and re-enter after rewriting the guest's state.
Abso-frickin-lutely insane.
Actually, no.
Real mode is your shiny new i7 quad+ core CPU pretending to be nothing more than an 8086 from 1978. It does that because upon initial power up, none of the data tables required for protected mode operation are setup. So there's a chicken and egg problem at power-up. You need the protected mode data structures setup, but they are not present yet, so you have to start somewhere. Real mode on anything 80286+ is intended as a way to provide a CPU that can run a program that can setup the protected mode data structures so that protected mode can actually be entered.
v8086 mode was introduced in the 80386 chip as a way to provide hardware support for visualizing multiple real mode processes/applications while remaining in 80386 protected mode. I.e., it was some amount of hardware support for what VMWare or VirtualBox provides today. The faulting into protected mode on illegal access is/was intentional. To remain "protected" the code running in v8086 mode can't be allowed to do things that might bring the system down. So v8086 faults those accesses over to protected mode, and the assumption is that there is a protected mode handler that performs the operation (or denies it) on behalf, safely, so the system can continue to chug along. The capture/rewrite state is the intended design consequence.
An extremely good book on the 80386/v8086 design and how to make use of it is: "Advanced 80386 Programming Techniques" by James L Turley, Osborne Publishing, 1988. A very easy read, and extremely well written (assuming you can find a copy).
Made me smile :)
>THE OS IS STILL CRASHING WHEN I PRESS A KEY. This continues for 2 days. Remember that now that I have remapped interrupt 1 to interrupt 33 and I need to update my IDT.
I have issues like this a lot (i.e. forgetting I've tweaked something and wondering what is causing weird effects). Any advice on how to avoid it?
Every time you think, write down a line or two delineating your current reasoning and your debugging info. When you make changes, highlight them to mark them as pending ("need to update IDT"). You can also categorize, tag lines/pieces, and so on.
When you're stuck just backtrack this file, whether it is a hour or two days from now, and you'll avoid all of this.
Use your version control, commit more often and use more descriptive commit messages, along with branching/merging strategies to avoid putting not-production-ready code you are unsure of to master.
And I'm guessing that the difficulty comes from the fact that she's building a freaking operating system from scratch... ;)
It's also quite jarring to read; I was confused if the OP was speaking of someone else. People often point out grammar or factual mistakes, I see your parent as doing no different.
EDIT: And it goes a bit further. The sole reason why we are having this discussion is because the author explicitly used a female name and her writings still got perceived as those of a male author. If there no name or just a generic nick, most people in here - myself probably included - would have assumed its a male by default.
I guess I see a blatant mis-gendering as not 'banal.'
It shouldn't be. There are two things going on here:
* Perception of our industry as all-male biases women against joining
* Default use of 'He' exacerbates this by unsubtly signalling that our community is all-male
So why flag female hackers?
* To try to combat the perception of us being an all-male community (we're not)
* To signal to women and men that both genders can operate comfortably in this environment
Why do we want more women hackers?
* We are ignoring more than 50% of the programming talent in the world
When in the tutorial by accident she called a woman by "he", she told that in doubt in the tutorial for Computer Scientists she calls people by "he" and in the tutorial for media scholars by "she", since because of the gender ratios of these courses of studies this behaviour maximizes her likelihood to be correct.
I'm confused by your characterization. Why do you think it is brusque?
I remember fresh out of high school following a few tutorials and writing a basic kernel & code for my old Pentium MMX and reading through the intel developer manuals for the 386 processor , and all the complexity involved that was inherited from the 8086 & 286 processors. Was a fantastic learning exercise... though I would suggest against using Pascal as the implementation language.
Writing an actual graphics driver is tricky - although there are some ways to get decent frame buffer modes (like VESA Bios Extentions), to do things really well requires thousands of lines of code before you can even get a pixel on the screen, and is different for every card.
It may be a bit easier now - you could look at the kernel mode setting code in the Linux kernel that exists now for Intel, nVidia (through Norveau) and AMD cards that didn't exist a few years ago when I was dabbling around in OS dev. Still, it would be a massively difficult task. You'd probably want to use the full Mesa stack and open source drivers that use it if possible, but writing the runtime to support it would be a big task.
The most fun mode was mode 0x13 from INT 0x10, with its fantastic 320x200 resolution and 256 colors. That one had the buffer at 0xA0000.
In addition you have to be using I/O ports if you want to move the blinking cursor and other related stuff.
> Writing an actual graphics driver is tricky - although there are some ways to get decent frame buffer modes (like VESA Bios Extentions), to do things really well requires thousands of lines of code before you can even get a pixel on the screen, and is different for every card.
This is particularly painful because you have to switch back to real mode to be able to use VESA and other Bios interrupts.
Thankfully it's not required if you're using a good bootloader like GRUB (which you should if you're doing an OS project, writing another bootloader is just more arcane hardware to work around). The multiboot specification allows you to ask for a video framebuffer from the bootloader, and you'll get a physical memory address where the linear video framebuffer resides. I bet UEFI has something similar too.
So these days, with the help of your bootloader, doing graphics on a bare metal kernel is as easy as using the text mode.
EDIT: Nevermind, that still required putting the string in a magic location.
Does that include things that are not in text mode, if I recall the name correctly? Sometimes I just want my tiniest netbook to display the normal 80x40 resolution but it keeps putting the characters in what I think is video mode
The "correct" thing to do here is set the console font to a larger one. If you are on a systemd-using distribution, see vconsole.conf(8).
Now, you could force a lower resolution, but that causes jaggy text, as pixels will be stretched unevenly by the graphics card. This can be done with KMS, or the vga= boot parameter.
The cycle was: 1) insert ioout to turn on speaker at code line X; 2) compile kernel 3) cause lockup 4) is speaker on or off? 4a) on, move ioout to a later code section; 4b) off, move ioout to an earlier code section;
Essentially, a binary type search through the code, using the on/off state of the speaker hardware to narrow the search. Turned out a lock needed to be one line later in ll_read_write_page.c (if I remember that file name correctly).
So, the answer to "how do you debug" is: "with anything you can make use of".
I do remember the excitement of getting interrupts to work - good times. Getting task switching to work was magic.
True emulators like Bochs are still potentially a win if they provide fine-grained debugging features. Bochs was nice because you could single-step through assembly and view contents of registers/memory quite easily. You could also attach gdb to it (although this was somewhat flaky.)
1) You don't need an OS to press keys and make them come out in the screen. An OS is kind of a library that an application uses to do things.
2) I believe what the article refers as "OS" is a small process manager. I don't know if it's preemptive, no mention of timer of any kind.
3) As far as I can see, this is not an OS but a ring-0 application that reads the keyboard using IRQs. IMHO an Operative System should at least supply filesystem/memory manager or process-manager services.
Hacker school alumni, those articles are awesome but please get your concepts right and at least learn the correct terminology or you will only confuse people. But I suspect those articles are vague on purpose to incite controversy.
Every OS projects starts out as a "ring 0 application" running on bare metal. An application which does nothing except handle interrupts and spew out numbers on the screen to tell that it is working. And most OS projects never really evolve a lot past that, because they are primarily intended as educational tools for the author and nothing else.
If you had a point, you could have made it in a way that makes you seem like a decent person, not an angry antisocial geek.
> This continues for 2 days
Imagine if my blog post would talk about my experiences with building an HTML page, and how it took me 2 days to change a <p> element to a <div>. Let's get serious here.
How long will it take the author to write a circular buffer to read and write characters to the UART? A month? That says nothing about the difficulty of the actual task.
You know, when it comes to writing operating systems, I like to think of this quote: "it's hard, it's harder than it looks, but it ain't THAT hard" (I first heard it when the commentators were laughing at Chris Andersen was struggling to dunk in an NBA Dunk Contest).
If you're interested in writing an OS, make sure your primary source isn't some blog post. My textbook was Tannenbaum's Operating System book, and it's a very solid book that I can recommend. Tannenbaum knows operating systems. He's not going to tell you how hard it is. He's going explain all the details to you and, when you understand them, you'll think how easy kernels really are.
Good luck!
Unless she was spending full time on it, it was not some bug that takes 2 days.
At the same time: If you've never come across some trivial, stupid little bug that has left you stumped for far longer than it should have, you're either superhuman (or have selective memory), or you're a total beginner.
Your dismissive tone is really offputting.
* re-using a loop variable from the outer loop in an inner loop (damn you `int i`!)
* putting a char[256] on the stack when the stack size was only 256 bytes and getting stack corruption (this one wasn't me, thankfully)
* signed vs unsigned code being passed between user mode and kernel mode causing problems (no idea why it was causing issues; i just made it all unsigned and the issue went away)
In hindsight, -Wall would have probably caught everything and saved a couple hours of "wtf" at 2-3 in the morning.
Often the simplest things are the hardest to discover.
> THE OS IS STILL CRASHING WHEN I PRESS A KEY. This continues for 2 days.
The lesson conveyed in your post is "if something is hard for you, keep your mouth shut about it, or someone will laugh at you." That is a fucking terrible lesson and it needs to be stamped out and destroyed. I'm still trying to unlearn it.
The lesson conveyed in my post should be that this author is making it look harder than it actually is, and if you want to write an OS, you shouldn't feel like it's an unsurmountable task. But by all means, stamp and destroy it.