Surely you don't need the phone's actual latitude and longitude. I can't imagine that the direction to a point infinitely (basically) far away will be different depending on where on this tiny planet you're standing. I'm sure just using the Earth's position (or any other position in our solar system?) would be sufficient to get a valid pointing vector.