"the CPU/GPU latency cost is going to be close to the original CPU evaluation time."

  Have you looked into HSA architecture that helps to remove this latency?  I think this is the direction Intel will move to in a few years.