GEM predates GEOS by many years. It was a product of Digital Research, and was started in the early 80s by Lee Jay Lorenzen who had worked (or interned?) at Xerox on the Star and wanted to bring those concepts over to commodity x86 desktop machines.
Lorenzen went on to create the successful DTP program, Ventura Publisher, which itself used a bundled copy of GEM.
The wikipedia article on GEM has a good chunk of history: https://en.wikipedia.org/wiki/Graphics_Environment_Manager
A 68000 port of GEM and GEMDOS (which itself was a kind of fork of DR's own CPM/68k) were the basis of the Atari ST operating system "TOS". Some of the tales of how that all went down are documented by one of the participants here:
http://www.dadhacker.com/blog/?p=995
http://www.dadhacker.com/blog/?p=1000
All of this an entirely separate effort from GEOS/GeoWorks, though in some respects the two efforts are similar in that they aimed to bring the full WIMP experience to fairly low-powered machines. But the philosophy was quite different. PC/GEOS was written in a lot of assembler, specifically for x86 (after they'd done something similar for the C64 in 6502 assembler in fact), specifically for commodity PCs. GEM was written in C with some amount of portability in mind (in theory), and ran on top of DOS, CP/M, or GEMDOS (I believe there was also attempts to get it to run on top of Unix). The first version of GEM was developed and ran on the Apple Lisa hardware.