it would take more energy to mine [granite] than you would get from it. There are only 3-10 part per million of uranium in granite.Have you read this?:
http://nuclearinfo.net/Nuclearpower/UraniuamDistribution
The Rossing mine in Nambia mines Uranium at an Ore concentration of 300 ppm at an energy cost 500 times less than the energy it delivers with current thermal-spectrum reactors. If the energy cost increases in inverse proportion to the Ore concentration, shales and phosphates, with a Uranium abundance of 10 - 20 ppm, could be mined with an energy gain of 16 - 32.
Extrapolating from that, the energy gain is 1.6 for every ppm of uranium. As you say, granite contains 3-10 ppm of uranium. Therefore, mining granite for its uranium would provide an energy gain of 4.8 to 16. That would be without improving current reactor thermal-efficiency at all, and without improving current reactor breeding efficiency at all (all fission-reactors fueled with a mixture of U235 and U238 breed to some extent).
Running a uranium economy off of such a low-grade ore would, of course, in the real world provide incentive to improve fuel-efficiency (thermal; breeding-ratio; etc.) of reactors (even thermal-spectrum reactors can have their breeding-efficiencies improved). Thus, in the real world, the energy gain from mining granite for uranium would likely be higher than 4.8 to 16.