Microsoft faced a very similar problem with Windows - Windows was 16-bit code, a mixture of C and assembler. How did Microsoft handle it?
Firstly, it isn’t necessarily hard to port 16-bit real mode code to 16-bit protected mode. It really depends on how much segment manipulation you do. If you treat segments as opaque values and hide operations on them behind some API, all you have to do is change the implementation of that API.
Secondly, you can create a 32-bit protected mode kernel, and then run 16-bit code under it: you can run 16-bit code under it directly (just mark some code segments as 16-bit and others as 32-bit); if you really need to run 16-bit real mode code, you can use virtual 8086 mode. This is essentially what Windows did, back in Windows 2.x/3.x/9x/Me timeframe
You can use thunks to mix 16-bit and 32-bit code so they can call each other. That way you can write new code as 32-bit, rewrite performance-critical 16-bit code as 32-bit, but leave existing non-performance critical 16-bit code as-is.
You can even reassemble 16-bit assembly to run in a 32-bit code segment, the assembler just needs to stick 66 and/or 67 size overrides before (most) instructions. It will bloat the code, but it might be an option in some cases.
Another option, is write a transpiler from 16-bit assembly to C, and then compile that as 32-bit. DEC did something similar for the OpenVMS transition from VAX to Alpha - they wrote a compiler for VAX assembly language which compiled it to Alpha machine code; the same idea was carried forward in the Itanium and x86 ports. Similarly, in the early days of 8086, a lot of 8080 assembly code was ported to 8086 using tools that read in 8080 assembly and wrote out the 8086 equivalent (e.g. Intel’s CONV86, Tim Paterson’s TRANS86, Digital Research’s XLT86)
So I don’t think GEOS’ technical debt was insurmountable - Microsoft faced some similar issues with Windows (even if they didn’t have it quite as bad). Various options were available to address it. Rather, I think the real issue likely was that GEOS couldn’t attract sufficient investment to pursue them.