“ With machine learning, you spent most of the time copying memory between the CPU and GPU”

- this is a sign that you are most likely doing it wrong. Yes, some operations are inherently bandwidth bound, but most important ones such as larger matrix multiplies (transformers) and convolutions are compute bound.