Their secret at the time? Making an actually good browser that offered features the incumbent didn't have, something Mozilla has absolutely given up on in pursuit of making a Chrome lookalike, up to the point of removing certain features (that power users - their only marketshare at this point - very much miss) and even implementing arbitrary restrictions on the add-ons you can install on Android (you have to use a complex workaround to install unapproved ones). The thing is, people who want Chrome will just use the real thing and have no reason to use a slower lookalike with poor compatibility.
Here's an easy idea to sell Firefox on the desktop to the masses: built-in uBlock Origin, out of the box. Nobody I know that has tried an ad blocker ever switched back, and if they do fall for a dark pattern and switch back to an alternative browser, ads will be a persistent reminder telling them to go back to Firefox. Actually give non-technical users a real, tangible reason to use Firefox.