I lived in Provo and Orem for a number of years while I was attending University, and I feel you've nailed some of the major issues. Life was drastically different there compared to where I grew up (and now live) in the Bay Area.
One on-topic aspect I appreciated about Utah was the quality of the Computer Science program at Utah Valley University. The chance to learn from experts like Chuck Allison [1] and David Heldenbrand truly expanded my mind and was an incredible opportunity. Really, all the CS faculty was excellent and commendably eager to do the teaching themselves rather than offloading the most critical parts to TA's. Given the high ratio of religious fanatics in the surrounding region, the school is a remarkably open and progressive institution.
As soon as I finished school I moved back to Palo Alto and would never consider returning to Utah other than to visit friends, go skiing/snowboarding, or go to hike at Moab or Zion National Park.
I definitely wouldn't recommend going to Provo or Orem to start a tech company.
To put things in perspective: Provo is 29 short highway miles from Eagle Mountain, which is primarily composed of polygamist compounds. I'm not saying there is anything wrong with being a polygamist; it was just a surprise to find out about it only after several years of living in Provo.
There is a reason that Provo and Orem often referred to as the "Utah Valley Bubble". It is an isolated community and if you aren't Mormon you will likely pretty strange when you discover how poorly you fit in with the homogeneous culture.
With that being said, there are some upsides to operating a business in Utah Valley:
You can hire high-quality engineers for pennies (or perhaps a quarter) on the dollar compared to other places. At one point I interviewed at Omniture, and they scoffed at the salary I asked for.
People who live in Utah Valley are also less likely to move away as much compared to other places, because their immediate and extended families tend to live nearby. This means that it is possible to expect to retain employees for much longer period of times than in a place such as Silicon Valley.
Finally: If you are into discriminating against people for their sexual orientation, it is completely legal to do so in Utah. There is no legal protection from sexual orientation discrimination in Utah.
[1] https://www.google.com/search?q=chuck+allison