That depends on the problem space. For some problems latency does not need to be an issue, if the problem can be pipe-lined then you can keep all your cores running at a high percentage of their theoretical maximum load even if latency is high.

For real time applications this is usually not an option, there latency > some threshold will kill your proposed solution.

You overcome the bottleneck by doing as much as possible on the GPU. Even things you don’t think are fully suited to it.