How DOS was able to use most of the 1 MB address space of the 8086 | Hacker News Reader

How DOS was able to use most of the 1 MB address space of the 8086

blogsystem5.substack.com

145 pointstodsacerdoti71 comments

It's a miracle emm386 managed to squeeze out more memory as UMBs than it needed to do its job.

Consider: It scans for holes in the memory map. It switches to protected mode, then v86 mode, sets up page tables, and is basically more of an OS than DOS, if you look at the CPU. It monitors IO ports and rewrites DMA access from random programs to translate the UMBs to their physical addresses. It implements VCPI to cooperate with e.g. windows and dos4gw. It switches A20 on and off when translation needs to happen.

And all that to gain memory from segments C000,D000,E000, if nothing else sits there. So 192K max.

Looking back, it's insane what contortions we went to, just to keep dos alive.

It's more like a hypervisor than a true OS, and VM86 mode was the first hardware virtualisation feature of the x86 line.

Windows/386 and the DOS-based Windows up to Me were similarly hypervisors for DOS.

It's a shame that VM86 mode wasn't able to be nested. I don't remember the specifics, but as I recall there are instructions that don't trap and prevent nesting VM86 within itself. I know Intel rectified this in later CPUs-- I believe that's the VT extensions.

The VT extensions also unfortunately didn't "fit in" with the VM86 model either, needing their own special instructions and data structures (and AMD and Intel also did slightly different things, further complicating the situation.)

sgdt/sidt/sldt are some of these instructions. Don't know if there are others.

They give linear adresses of the processor's descriptor tables. Code running with paging enabled can't even read these locations, so they are pretty useless for user mode. But they allow user mode to get forbidden info from the kernel, and virtualizing lgdt/lidt/lldt becomes impossible because non-privileged code can see the values given to the cpu changed, and a save state/restore state is impossible under virtualization.

Under newer processors, flag CR4.umip makes them trap, so virtualization becomes possible.

> It's more like a hypervisor than a true OS

True.

This is why I've suggested an inverse EMM386 to the FreeDOS team: a 32-bit app that has a single purpose -- start 1 DOS VM.

That is the only way I can imagine to create a form of DOS that could be booted on UEFI computers, and without that, soon FreeDOS will be confined to VMs and old kit.

The idea is, start a very very simple 32-bit binary that creates 1 V86-mode VM, maps some DOS resources into it (disk, screen, keyboard, mouse, etc.), then loads the DOS kernel into it.

In DOS a stub EMM386.EXE could create some UMBs and LIM EMS if you want it.

I doubt it will ever be possible to support sound cards, network cards, etc. this way but most modern ones don't work on DOS anyway so nothing of importance is lost.

DoubleDOS [0] did this before VM86 mode was a thing and worked on 8086 CPUs.

[0] https://en.wikipedia.org/wiki/DoubleDOS

You know that there were loads of such things, right?

DESQview, IBM TopView which inspired it, Windows 2 itself, Windows 3 in real mode, DR Concurrent DOS and CDOS 286...

Zortech C++ had a special "handle" pointer type that enabled access to expanded memory, where the work was done by the compiler:

https://www.digitalmars.com/ctg/handle-pointers.html

There was also a VCM system that swapped code to/from disk in a manner that was very much like virtual paging.

https://www.digitalmars.com/ctg/vcm.html

The big achievement of DOS and the 8088 was relocation during loading. For .COM files, this was easy: the program was segment register relative, you pretty much had virtual memory. For .EXE files, MS-DOS relinks all of the absolute segment addresses during the load process.

So this means you could have multiple programs loaded at once: often done with TSRs (terminate and stay resident programs), but it also allowed things like shell windows in Lugaru Emacs (Epsilon) to exist. Languages like Turbo-C had system().

Also the OS was well decoupled from the application programs (it could be independently updated). This was not true for many of the other home computers of the time, but some pulled it off: all CP/M systems and TRS-80. I think the main exception to this was the video system- many programs made assumptions about the address of the screen buffer.

To be precise, most computers had only a monochrome or a color card, but nothing prevented you from having (and using) both.

I remember a debugger that gave you dual screen on a 8086 by using the monochrome for text/debugging whilst the color screen was for the program.

SoftICE was amazing at the time.

I heard modern OS protection mechanisms prevent debuggers do things like SoftICE did. Is it true? Where can I grab more information?

I don’t know about that sorry. It did let you suspend Windows totally, not sure how different that is than today. I had to take over responsibility for a device driver in windows 95/98 era and SoftICE offered a better debugger than microsoft for the task under Windows at that time.

Windows kernel debugger also lets you to totally suspend Windows. You can configure it to connect over ethernet.

Designing a debugger like SoftICE is unnecessary now: modern processors support virtualization, which means you can run the entire OS inside a VM and debug it from your hypervisor.

Absolutely. I still remember the disbelief when I realized that you could observe Windows 95 booting. It hooked into system before the OS.

the microsoft debugger back then allowed the use of dual screens

I have always strongly suspected that this is how print debugging got cemented into the ethos.

You had developers using it as a power feature.

I did that with Turbo Pascal in the 90s. Having your app on the color monitor, while the debugger was on the Hercules one was amazing.

Ooo. I do not know that. … many decades too late.

If I remember correctly, even with a color card, applications switching to a monochrome video mode was possible if a bit rare. Creating UMBs in vram segments A000, B000 or B800 was always considered a suicidal configuration, and EMM386 would not do it unless forced.

If I see the man page http://www.manmrk.net/tutorials/DOS/help/emm386.exe.htm the vram segments are not allowed for the FRAME= and Mx options, rang A000-BFFF is forbidden.

The whole confused and confusing epoch of "expanded" memory versus "extended" memory proved that Intel processors were designed by the Devil himself.

> were designed by the Devil himself

a.k.a. Intel, who also designed USB "Full Speed" vs "High Speed"

("Full Speed" was the maximum speed of USB 1.0, 12 Mb/s, which also had "Low Speed", 1.5 Mb/s which is used by keyboards and mice. "High Speed" was introduced in USB 2.0, and is 480 Mb/s)

Joke's aside, I have empathy for the pain of naming in protocols and systems that get extended and remain backwards compatible. Intel was the king at this.

They should have called it Ludicrous Speed.

Cant wait for plaid speed usb in a couple of years

What are you doing?

Adjusting your bits. You crashed and they... shifted all outta whack. There.

(BTW this is a paraphrase of on my favourite gags from the early Steve Martin film Dead Men Don't Wear Plaid for reasons that I hope, in context, are obvious.)

Is that terminology coming from the CPU manufacturer or rather from the DOS makers?

Either way, confusing stuff.

My other pet peeve about Intel is, why the heck would they make a paragraph just 16 bytes ? Why not 256 ?

The 8086 was a stopgap, to have something to sell until the 432 was finished. Maybe they wanted it to be an evolutionary dead end, and 16MB address space was too dangerous? Notice also how the 80186 was PC incompatible.

I suggest you take a look at DOS extenders at some point. They were brilliant and the best was DOS/4GW.

The article mentions those at the bottom: "And that’s it for today folks. I intentionally did not touch on DPMI—the technology that truly allowed DOS applications to break free from the 1 MB memory limitation—because I’m saving that for the next article. So, make sure to come back for more!"

[Original author here.] Exactly! That (DPMI) is the thing I originally wanted to talk about but I realized I had to clarify many more thoughts first! And I still have to do the research on DPMI to be able to talk about it :)

It would be interesting to understand the Quarterdeck products and how they fit together: the separation between QEMM and QDPMI, and how DESQview worked to enhance DOS with multitasking.

I was almost amazed when instead of launching windows 3.1 running win.com one could run Borland DPMI host (that they used in their own 32-bit compiler toolchain), that stayed resident offering its API, and then one could run windows' KRNL386.EXE, that hooked with the DPMI host and booted windows.

What benefit comes from this?

None concrete I can think about, apart from understanding better the innards of windows boot with a whacky experiment. DPMI was an open spec, so this validated that in principle you could write your own DPMI host to make windows think it was running on top of a protected mode whose view to clients you could control and customize. But again can't answer to what avail.

Zortech shipped with a compact and fast 32 bit DOS extender called DOSX. It worked like a champ.

https://www.digitalmars.com/ctg/dos32.html

And supported the 286 DOS extender from Rational Systems.

Oh yes, that's right it did. I do remember playing about with that.

CauseWay DOS Extender was released open source to public domain in 2000, and the source is on GitHub, if you're curious about the implementation. It supports builtin EXE (LE) packing as well.

There were many of these.

An impressive suite was the stuff from QuarterDeck. Their DOS extender was called QDPMI, and used the QEMM memory manager.

Quarterdeck DESQview, built on QEMM, truly extended DOS by turning it into a kind of multi-tasking system, allowing users to be more productive.

It was known for its high application compatibility.

I used DJGPP in the early 1990's, and the GO32.EXE extender, which did the job.

My 1990s multi-node BBS ran via DESQView/x

That company produced a number of excellent DOS survival tools. QEMM memory manager was an absolute must to get games working.

DESQView was my daily driver for a number of years. I even ran Windows 3.1 in a DESQView window.

Does anyone know how QEMM worked exactly? I thought all the tricks would be known by now but no other implementation comes close to it.

Not all of them.

But the rival 386Max is now FOSS, so if you are really really curious you could take it apart and study it.

https://github.com/sudleyplace/386MAX

Closest you will get for now, I fear.

I think I read that Symantec lost or erased all the Quarterdeck source code after the acquisition. :-(

DOS/4GW was memorable because it advertised itself while loading games made with it, usually late 386/486 era 32-bit games from before gaming on Windows had its act together.

Phar Lap was more ubiquitous throughout the 286 and 386 DOS days. Microsoft distributed a light version of it with their compiler for a while.

Pharlap 386 is mentioned on Links386, although its flies on by, and special->about

Pharlap's TNT was way more exciting to be honest.

I'm still trying to find the first early extenders though from 1987/88 no leads though.

They also had DOS/16M which targeted 286 protected mode. The company I worked for used it when moving the Clipper programming language to protected mode.

DOS extenders and Watcom C/C++ were heaven-sent after spending years targeting real-mode with its segments and offsets.

I do wonder if it's possible to write a DOS extender for 64 bits.

The big problem is that 64-bit x86 long mode removes the V86 mode that made DOS 386 memory managers possible.

This is why the DOSemu project has been doing a multi-year rewrite: to create a new, full-VM-based DOSemu2 that can run DOS without emulation on x86-64 machines.

https://github.com/dosemu2/dosemu2/wiki

I feel old because I still remember the memory management game very clearly

Yes. I still remember back in the day we used PC Tools to modify hot game data and bought pirated games from corners of alleys.

I feel even older because I’m starting to forget.

Helix Netroom[1] is missed in the article. Hope it will be in next part of series.

[1] https://en.wikipedia.org/wiki/Helix_Netroom

If memory serves, I had a neighbor who bought that 4MB Emulex card.

It was for an 80286 clone that had a 10MB HDD. His rational for buying this card was "I can make a big RAM drive out of it and run everything from memory"

I can't recall the price of the thing, but I want to say it ran in the thousands. Couldn't run most games on it because of copy protection so we never got to take advantage of it. In fact, I don't think the extra memory was usable at all in most software -- I recall we couldn't run a game on his machine due to having 512MB of onboard memory, but it had no issue in my ancient 8088 with 640K installed (in bizarre CGA colors).

CGA was the worst. EGA and VGA were huge improvements because CGA only had 4 ugly colors, while EGA had 16 (good), and VGA had 256 (much better). VGA also supported pallets I think (a way of allowing an application to use 256 colors out of 16 million supported colors).

Nitpick: the VGA DAC has 6 bits per RGB component (18 bits total), so each of the 256 colors is chosen out of 2^18 = 262,144 colors, not 16 million.

> CGA only had 4 ugly colors

16 colors, technically. But only four at a time, and three of those were selected from two preset palettes, both of which were hideous (one green/red/brown, one cyan/magenta/white).

And EGA had 64, because while its display output pins where still TTL, there was no longer a single intensity bit (RGBI) like in CGA but individual intensity bits wired for each color component (rRgGbB)

And yet... https://int10h.org/blog/2015/04/cga-in-1024-colors-new-mode-... (this was resubmitted here a few days ago)

You nailed it with "4 ugly colors"[0]

I remember the palette choices in CGA were downright bizarre; they'd be 4 colors that you couldn't possibly use to make anything look right. At the price of the card and the display, you're like "wait ... that's it?"

[0] Ignoring all of the interesting things coming out of the 8088 demo scene, nothing that I'm aware of from the 80s was capable of anything close to that.

I was avoiding EMM386 as much as I could on my slow 386SX25: there was a huge performance hit.

There was the Unreal Mode aka Flat mode. With few lines of assembly code, an memory allocation using HIMEM.SYS, it was possible to access all the memory: the segment limit is 4GB instead of 64KB in this "mode". Of course it was not compatible with Windows, so the unreal mode was more for fun than anything else.

https://en.wikipedia.org/wiki/Unreal_mode

Ehh! I wonder, when somebody remember, ms-dos was NOT first to use such tricks.

- All these stuff was REimplemented on PC IBM's technology of overlays, created as marketing technology, to widen line of IBM mainframes, as top models have enough RAM to work with programs directly, but cheapest models have to use something like virtual memory and swap on magnetic drum, to use same software.

And sure, when IBM PC achieved wide adoption, people also sought, some things could be done on PC, what before done on mainframes or mini-computers, but much cheaper.

I exactly seen revolution in approx 1993-1994, when prices on RAM dropped and become possible to equip PC with 1M or even more.

Unfortunately I myself believed, to talks of older colleagues, that all these PC tricks are not serious, and IBM is strong and will return with something much more powerful :)

I even not imagined, about current 64-bit PCs and what opportunities they opened.

Thank you for this article. It reminded me how I was profitable in DOS memory optimization. To run game like Mortal Kombat, Doom, Dune, Dune 2 or others I got very good in optimizing. I remember now how I wrote my own config.sys and autoexec.bat files or run MemMaker. This is a piece of history for part of us

I landed two jobs in the 1990s on the strength of my DOS memory-optimsation skills.

I didn't use Memmaker, or Quarterdeck Optimise or anything. I did it by hand. I got as much free base memory, with shorter more readable config files, and they were more maintainable.

When I worked on my Commodore 65xx computers, I intimately understood their hardware architecture, the Page Zero layout, the general layout of RAM/ROM, PEEK/POKE registers that mattered, CPU operations, and I had a general knowledge of assembly language. And most of this was documented in the user manuals themselves; the only other stuff I read was "Compute!'s Gazette".

When I received my first IBM PS/2 in 1989, I hardly knew any of this stuff. I didn't know any x86 assembly, I didn't know how DOS laid out memory or how the INT routines accessed devices. My computer overnight turned into a black box. I didn't program it in BASIC, I just loaded games and applications on it.

This sort of system opacity drove me rather crazy, and soon I was in college studying Unix and systems architecture again. I didn't waste any time deleting DOS/Windows and replacing it with Minix.

To be fair my background is similar but the system isn’t really more opaque. I remember having the same books for the PC as (in this case my slightly older mentors) had for the C64. Undocumented PC and Undocumented DOS stand out. However you mentioned games , I too was distracted with such things (like the ABCs, alcohol, babes and cars). And had a nerd break until the web era. With maybe a little play around with VGA graphics and DJCPP or whatever it was called along the way.

I thought I was a MS-DOS 6 expert. But I had never heard of MemMaker. Probably because when it appeared I was already an expert at tweaking CONFIG.SYS and AUTOEXEC.BAT for gaming (I made a boot floppy with menu for each game that setup memory and launched the game), and MemMaker wasn't compatible with the menu system.

Reply on news.ycombinator.com