I'm not an expert, but in my experience working with CUDA/OpenCL/OpenGL, GPU latency usually comes down to data transfer and reduction operations (like max or sum). If you are okay with some kind of double-buffering, you can usually "hide" those latencies.