IMHO WinCE 6.0+ were pretty nice to work with. It could scale down to a system image that was just a couple of MB in size, and scale up to a full blown miniature version of Windows.
Prior versions of CE were a bit janky sure, but remember the sort of hardware it was meant to run on. Having a limited number of process slots isn't the worst idea in the world if you are targeting just a couple MB of RAM!
I'd say that just like any other large legacy code base, CE had some good parts and some bad parts. Everyone on the (6.0+ at least) kernel team knew their stuff, as did the media codec team. I know the C Run Time for 6.0+ was good (I was on the test team for it ;) ).
The lack of a concept of a current working directory was interesting though, of course no smart phone today has such a thing (and we'd think people to be crazy if they requested it!) but the idea was so baked into parts of the Win32 API that it was a fair bit of a porting pain for developers.
Drivers were always a problem for Windows Mobile devices. I actually owned some rock solid high performing WinMo phones, but they were few and far in-between. My Motorola Q9h is still the best smartphone experience I have ever had, it felt /natural/ for me to use, I cannot say that of any other phone I've used since. Also it was lightening fast on just 96MB of RAM. WinMo crawled at 64MB, but it flew at 96MB!
As an example of what it was like from the inside trying to make everything work, when working on WinMo6.5, we couldn't get our OEM partners to give us source access to their keyboard driver for us to make fixes for them to our test devices! (You ever try testing software when you cannot get hardware that it runs on? Ugh) They considered their keyboard driver to be "secret".
These types of problems were a large part of the reason behind the decisions for MS to write the majority of drivers for Windows Phone. When they wiped all the code, and started out with a brand new kernel, the decision was made to limit SoC support to ensure platform stability.
And that there is the kicker about mobile development. Every single new platform is a whole new adventure when it comes to software stability. Because so much changes when platforms iterate, you have a lot of drivers that need to be written.
Traditionally on PC we've only had a handful of chipset manufacturers, and the field has shrunk considerably. But in the late 90s I remember certain VIA AMD chipsets having known stability issues (a large part of the reason why AMD went to making all their own chipsets and not relying on 3rd parties anymore), and more recently even Intel has not been immune to bugs in their chipset drivers.
Now that the mobile field has consolidated to just a few chipset vendors (although this is changing with a flood of Chinese chipsets finally reaching awareness amongst western companies) that iterate on a good schedule, well at least phones don't just randomly reboot.
GPU drivers are still crap though, I'm not sure if the Windows Phone team still writes their own, but they sure as heck did for WinPhone7, there've been multiple posts on HN about Smartphone GPU driver bugs, I wish someone would test out WinPhone for those same issues! :)
tl;dr Aging embedded OS kernels don't behave well when scaled to 15x what they were intended to do. Just when WinCE started to get really good (involving a complete kernel rewrite) it was de-emphasized by the company.