Mozilla really just got caught in between being an OSS organization and working with proprietary phone companies, carriers, baseband developers, etc. We were used to working transparently on wiki, Github and IRC, and suddenly had NDAs and confidential code that partners insisted on. There was a large culture clash, and the effort both took a bunch of momentum from the browser side and ended up putting unique technical constraints on Gecko and the Firefox Browser, both in terms of necessary functionality and on the development cycle.
That last could be a killer because you couldn't do anything quickly as everything had to be tested at length for interoperability, etc. FxOS was usually pinned to something older than the browser development (consider the security bug implications there) for extended periods of time, and teams had to shuttle fixes back and forth. Given that some of the changes were basically hacks to get the phone going and had no benefit to the browser, I'm sure that caused a lot of friction.
Partners also wanted to see formal processes guaranteeing their stake, and that was at odds with the more "steer don't plan" way Mozilla tended to run. The people that were brought in that were more familiar with formal process and could make partners happy weren't as familiar with Mozilla's community or culture, and that also churned things a bit.
Basically, it was a perfect storm of issues to deal with--Brendan leaving was just one of them, and possibly the least important. Morale wasn't awesome, and the window of opportunity for a new smartphone OS was closing--if it was even open in the first place.
While there was some talk around shifting to feature phones as a focus/market gap (basically what Kai has done), it was very late in the game. After a point, I think a decision was simply made to cut bait and get back behind the browser again. Quantum was the (IMO very good) result of that focus.
Honestly, Mozilla did as well as I'd expect any organization of its type and size--which is not as well as I'd expect a greenfield organization tightly concentrated on the goal to do. KaiOS is that latter thing.