"Would a mobile network provider allow an open firmware to connect to their network?"

They would likely not be happy about it. I assume it would be something like the relationship between tivo and people running hacked tivos ... a cat and mouse game that a bit of effort can keep you ahead of.

BUT, instead of trying to bake up a ground-up open firmware, which would be wonderful but VERY difficult, better to just pick an existing firmware, like osmocom did with the (very old) calypso chipset. You would pick a firmware with very wide market penetration so that it is easy to get handsets, and then try to pick one that was easy to hack.

So ... whatever very widely adopted firmware has the most holes in it.