OWON seems to have written their own RTOS for the SDS7102. Except for a few strings referring to yaffs (which they don't actually seem to use) I didn't find any strings in the firmware mentioning any of the RTOSes I know about.
And Linux isn't that slow to boot if you are a bit careful. It takes about 10 seconds from when I press enter to start downloading Linux using openocd until I can log in with ssh. 3 seconds of that is the JTAG download and a second or two is the time it takes to get a DHCP lease over the network.