Unless you have like 64 cores, I don't think you will be a GPU. And yes, you can use multiple GPUs in parallel to train.