We live in an age of libraries like cudnn which are vendor provided/optimized, for 99.9% of ML applications you won't need to write custom CUDA code. Further as I mentioned earlier with Intermediate Representation languages like TVM you are guaranteed to get the best underlying constructs across different computing architectures.

Julia might still be good enough MATLAB replacement for Computational Simulation style tasks, but its clearly not suited for Machine learning.