I think this highlights the fact that it would be much easier to start with open hardware and install something intended to be fully open like Ubuntu Touch on it. Unless you do that, you're going to need to reverse engineer every device driver for every piece of phone hardware – you'll never finish and the user will always be cut off from some percent of their phones features and the rest might not work correctly.
I guess that a pleasurable user-experience might not be the end-goal and maybe this is fun because hacking.