Easily one of the most impressive passion projects I've seen here in a long time. Out of curiosity, why did you opt to write your own C compiler?
Did a little testing myself.
Tried it on some bare metal, looks pretty good.
There were two phases, wouldn't fit in a single comment.
Wrote RetrOS-32.img to a blank SSD and it boots in CSM/Legacy mode.
That's a nice milestone right there :)
This was on a tiny Lenovo desktop about as old as an x240. These are low-power, run on the same Thinkpad A/C adapter, so there are some similarities to a laptop.
But the cursor wasn't very responsive and wouldn't come down below the very top rows either so could not sign in. Using mainstream wired mouse & kybd. (The PC has 8GB memory, in the login box it showed the nominal 15mb, but extended memory was a negative number, about 8GB itself.)
Then I remembered things like the x240 often have a touchpad that is recognized as a PS/2 mouse, not USB. Moved the SSD to an old tower that has a PS/2 mouse and kybd and then could sign on. Cursor was still quite a bit sluggish, but it worked. This one has 12GB memory and the reported extended was about 464mb so it looked realistic in the login box.
It boots! It works nominally! It's some kind of Golden IMG already, so congratulations!
Back at the Lenovo I do boot to DOS from time to time, and I enable Legacy USB in the BIOS so I can use the USB mouse for DOS which had no concept of USB. After booting DOS, if I want to use the mouse I still have to load DOS mouse drivers, just like the 1980's but this firmware setting fools the old pre-USB DOS mouse driver so it will handle the USB mouse & kybd that people are using now. This setting has been there for ages since the PS/2 sockets on PC's started becoming scarce. Checked the BIOS, and Legacy USB was enabled the whole time.
However, on this Lenovo there was an additional setting, for "USB Virtual KBC Support". I don't know if I ever saw this along with Legacy USB settings in one BIOS, up to now would have assumed they were both the same thing but using different terminology. This setting was not only disabled by default, the text says this one auto-disables once an XHCI driver has been loaded. Then definitely found it to auto-disable after booting Windows.
Enabled Virtual KBC, booted to RetrOS and the USB mouse could log in. More sluggish cursor than on the PS/2 machine though.
Plus there were some untyped characters that would sometimes appear when logging in, or admin would duplicate itself so username said adminadmin after you typed in the password below.
Still has a huge negative number in the login box for extended mem, that was not a show-stopper, mem appears normal when looking at the system stats once logged in.
I would now imagine this Virtual KBC setting emulates how mouse & kybd are presented to a VM?
Interesting finding that DOS does not require this setting for a USB mouse to work on bare metal, but RetrOS does.
Have not put it through its paces thoroughly, focusing on the bare metal aspect.
Key sectors containing things like the Partition Table, or a bootsector (Boot Record) are also supposed to be ones that do not change whether or not there is an OS (or any files) in the filesystem or not.
Break that down a little bit, a partition table should not change whether or not the partition(s) are formatted or completely blank. Not supposed to matter whether you are going to use the partition for DOS, Windows, Linux or whatever.
Then a bootsector doesn't even exist until its partition is formatted to begin with, then it reflects the "geometry" that the formatting process seemed to think was best for that filesystem based on the information it had available.
Neither of these should change after that, and be completely independent of whether there are any files in the filesystem.
It's probably best to build on as firm a foundation completely consistent with regular old PC's and it really looks like this is going pretty well using FAT16.
Once you have copied the IMG to a SSD, you could look at either IMG or hardware in DMDE and see the same thing. You do have to kind of figure I checked the IMG for potential bootability before I did anything on bare metal ;)
It is intriguing. There were other anticipated possibilities for an IMG to be bootable, like floppy mode or "large floppy" mode, which start with only a bootsector located at sector 0, no partition table is needed since there is never more than one partition.
Sector 0 right now is reminiscent of a FAT16 bootsector, looking at it in "FAT/FAT32/NTFS Boot Record" Mode, it displays real well. With FAT, you will see there are two different entries for Total Sectors.
By the time FAT16 came around, the upper Total had already been deprecated. You are using it for a full total of 65535 sectors which works so far but it should be in the lower Total Sectors entry for this template on FAT16. The upper entry should be 0 instead.
Then interestingly, looking at sector 0 in Partition Table mode, you do have a (un-needed at this time) partition table, in the usual position within the final bytes of sector 0. The 65535 total sectors are independently designated there (same total as above in the bootsector code that takes up the bulk of the bytes in sector 0 above) but there is no CHS geometry in place, and DMDE flags a "Relative Sector" of 0 as incorrect because it's expected to always be something other than 0. There's a number of structural features that can cause DMDE to flag them other than this too.
With sector 0 in DMDE you hit Editor > Partiton Table and all you're seeing interpreted is the last dozens of bytes of sector 0, it's only meaningful if there actually is a partition table in sector 0. There would be a maximum of 4 primary partitions in place. You can change this part all you want without having an effect on the code existing in the earlier bytes of sector 0 (such as an MBR ordinarily). But if you've got partitions the partition table better be sensible or it could fail to point directly to where the target code is.
No real partitions at this point so you can just go into Edit Mode and zero out the top primary partition line:
80 0E 0 0 0 0 0 0 0 65535
You have to hit File > Apply Changes. Then the partiton table is completely gone. And it will still boot just the same.
Random finding; there's a gfx_destory_window in sector 284 that looks like a typo.
Nothing in the OS should ever be hardcoded to rely on specific sectors, instead everything needs to be relative to the "Hidden Sectors" in the FAT which determine the offset which is the actual bootsector location for a primary partition.
The bootloader needs to be independent, and the OS itself needs to function regardless of what sector its partition starts on.
Start with a zeroed SSD, open it in DMDE, hit Editor > Partition Table and there is nothing but zeros as expected. Type in this, analogously to the above:
80 0E 0 32 33 4 52 48 2048 65536
Also manually put the required AA55 signature at the final 2 bytes. Apply Changes. You have just partitioned it a very mainstream way.
The Relative Sector value shown is supposed to be incorporated into the bootsector at that position as the Hidden Sector offset, which it uses to find its way into the filesystem.
Therefore ideally after you copy the IMG to the SSD starting at sector 2048 (rather than starting at the now-proven, wonderful sector 0), it would be like any other OS on any other partition, with your functional bootsector right there on 2048 where the partition table expects it to be, the only thing to change is your original "Hidden Sectors" of 0 so that is what to edit.
In DMDE Partition Table view, double click on the 80 0E...2048... line and it will take you to sector 2048 as expected, plus autointerpret it as a bootsector, displaying the correct template for FAT bootsectors.
Edit > Edit Mode and where it says Hidden Sectors 0, carefully change that to 2048. Drive Number is supposed to be 80 most of the time too. Apply Changes. The partition should be self-contained now. All you should need to do is boot to it.
You will still need a Master Boot Record (MBR) within the first 440 bytes of sector 0 as usual, which can interpret the partition table and jump to the designated Active partition whose bootsector at 2048 will take over from there. Regardless of whether the OS was originally "installed" to a volume starting at 2048, it needs to be able to work after recovery to a different sized volume which starts at something different like sector 512000 for instance some other time too.
I copy a current MBR from a working bootable MBR-layout USB, bytes 0-439 to the SSD.
And we're off to the races on a real partition.
The success was however, limited. After choosing a resolution, no more display :(
There is mostly no code yet in the 65535-sector 32mb volume, only sectors 0 - 3097 in this IMG are nonzero so far. This is normal at this scale.
Getting it going does seem to depend on specific code always existing somewhere in sectors 1 - 3097 rather than the OS relying on the Hidden Sectors offset within its own bootsector to find that same code located further beyond 2048. This is easy to have happen since hidden sectors has always been zero. VMs tend to abstract this since they are not on real partitions, and they don't always want to make it easy to move to bare metal anyway.
However there is really nothing wrong if the specific bootloader for this OS is permanently stored outside the filesystem at this time and/or at a hard-coded address for the time being, Grub did that sometimes so it would be outside the filesystem and not touched by DOS or Windows normally. I wanted to see if the OS itself can be easily relocated as an IMG to any partition within reach of FAT16. Formatted or not the IMG will overwrite what was there so you get your familiar format parameters and the only thing you're supposed to need to edit is the hidden sectors to match where the partition actually starts when it is different than original. But that didn't quite do it.
So next, additionally copied IMG sectors 1 - 2000 to the SSD starting at sector 1 of the SSD. (Didn't want to overwrite sector 0 with its new partiton table & MBR, nor the other good code now starting at 2048, which didn't leave enough room for all 3097 sectors of data in the IMG to fit in that space any more.) Incidentally, 3097 sectors is only about 1.5+ mb in the whole OS at this point, if it could be trimmed to 2880 sectors it would fit on a regular 1.44mb floppy. Partitons-R-not-us at all for that. Well, you've pretty much accomlished that and more on USB so it really is something to be proud of.
Sectors 1-2000 are completely outside the filesystem as I have it partitioned now.
Now it works, regular Windows MBR on sector 0 finds the Active partition like normal, jumps to 2048 as indicated by the partition table as expected, where it first encounters your bootsector/loader? code and proceeds nominally from there. No more failure of display. Any dependency on code being located in the original dev sectors 1-2000 is fulfilled now because a copy of it is there too, regardless of it being outside the current filesystem which now starts at 2048.
Appears to work with only sectors 1-1000 in place too. I would maybe leave the bootloader in that position and make sure the OS can do its thing starting at 2048 (everything relative to the hidden sectors value within your bootsector, at the time) without relying on any sectors below 2048 after the kernel has loaded.
At this point I should be able to reformat my FAT16 SSD partition using MS-DOS, which will not only clear off the RetrOS, but also rejigger the parameters seen on the template to be completely conventional, I would expect quite a bit different since yours are not very mainstream values. No problem if that's what your OS likes right now. Then I could easily copy io.sys, msdos.sys, and command.com (in that order can be best) to the freshly formatted partition and DOS would boot fine in its familiar landscape, never touching your code in sectors 1-1000 since that's not in the filesystem and not routinely accessed by DOS. When I was done with DOS, then I could copy your IMG back starting at 2048 again, and it would boot right up, without any experimentation, since the code it occasionally depends on is still there somewhere between 1-1000.
So you're ready to switch over to bare metal like this and tie up the loose ends in the real world!
In the long run you want your own simple bootloader to be a file on your filesystem that your bootsector simply seeks after the standard generic stuff jumps to your bootsector, and never depend on anything in earlier sectors than your bootsector after that.
Whatever OS you put on a partition, fundamentally it should be able to exist as just a bunch of stored files unless you boot to it.
Another milestone will be when you can just copy the files of your OS to a regular FAT16 volume that was formatted using MS-DOS itself. Do not skip this step. Even 20-year old Windows XP FAT16 is no substitute, and Linux is also non-ideal. Plus different sized partitions than 32mb will need to be taken in stride. The only thing that needs to be special is the bootsector itself, mainly the part that seeks your OS bootloader file specifically.
Use the mainstream old generic stuff to lead to your bootsector, use DMDE to craft a custom bootsector easily for any partition when needed without having to write your own Format routine, and concentrate on the OS functioning well within its own volume anywhere like the best of them!