I can't tell whether or not Xenix was available for other M68k platforms than the Lisa. The Lisa had some sort of MMU. The SUN 1 also had a custom MMU for the 68k. As long as you're content with dealing with the restart problem either by disallowing it (crashing any process that you can't help by triggering a bus error for - optionally minimised by doing paging while halting the main CPU like the Apollo's did) or other awful hacks, you can still do some things with the MMU with the 68000 too.
Here's details about the Lisa architecture that includes bits about the MMU and how it's handled in Lisa OS[1]. While it doesn't describe how XENIX did it, it gives some hints about which options might have been available to them:
> Although instructions in the MC68000 processor are not generally restartable, we have empirically determined that the four instructions that access code segments, JMP, JSR, RTS and RTE are restartable. When one of these four instructions attempts to reference a code segment that is not currently present, it causes a bus error which traps to the OS.
It would have been nice if this used compiler assistance. As only trapping those would mean ensuring that you insert one of them on every segment boundary if you want to be able to cross it, and also ensuring that you don't have any BRA/Bcc (relative branches/jumps or DBcc (relative branches/jumps with register decrement) instructions crossing segment boundaries. If XENIX did this too, then it'd allow relatively painless paging of code segments. It wouldn't be very hard do do, but give a strong incentive to try to pack functions into segments so that no basic blocks crosses segment boundaries.
But it seems the Lisa's default OS required programmers to explicitly split a program into different code segments:
> The division of a program into these named code segments is dictated by the programmer through commands to the Compiler and Linker.
No idea if XENIX also required that, or if they tried to be smarter about it. It wouldn't seem that hard to at least have logic to allow the compiler to automate it, though I'd guess some might want to explicitly control to be able to trade off amount of memory pinned at a time vs. amount of jumps between code segments (potentially triggering paging) given that the maximum code segment size of 128KB was a rather sizeable chunk of a machine as constrained as that (maximum addressable memory of 2MB)
> Because the instructions that reference data are not all restartable, the system does not do automatic swapping of data segments. The memory manager must swap in all data segments needed by a process before the process is allowed to execute. However, the OS gives programs the ability to unbind data segments that are not needed in the memory while a particular part of the program is executing.
You could presumably still allow memory to grow with brk(), but trapping SEGV to decide when to not so much. Though maybe - a hack occurred to me just now (no idea if anyone did it) to allow some patterns like that work: Have the MMU trigger a "soft" error condition if you access within X bytes of an unmapped page, and trigger a normal interrupt in that case. A normal interrupt on the 68000 is triggered immediately after the instruction has executed, so as long as a single instruction doesn't touch memory both within and outside mapped data segments you can trigger the pager when it gets "close enough". That's enough to e.g. make the stack auto-extending, or to allow sequential writes that approach and cross a page boundary, as long as the applications are ok with SEGV's not being triggered precisely when a page boundary is crossed.
You'd still not be free to page data segments out. But you could free stack segments pretty easily with some minor compiler assistance to communicate lower stack bound now and again.
Your idea of only issuing restartable instructions might work, with the caveat that the M68k is exactly the wrong architecture to have to worry about that on given how heavily it relied on various complex addressing modes to make the code compact... A lot of common instruction patterns would become unviable.
[1] https://www.researchgate.net/publication/2996607_The_Archite...