I transitioned to what's now referred to as ML engineering 11 years ago, as a programmer I started working on scripts doing all sorts of ETL (in Python - was my intro to the language) and handled datasets saving and loading for training. I managed models serving (we were using Theano at the time) in prod via REST APIs. Also worked (and still do) on writing model architectures with DL experts and I can say I still don't have solid understanding of the maths but I sure know how to manipulate matrices and write ML layers/models when working with an expert.

There are other aspects to the job (like dispatching experiments) but the point is that I was able to bring value in all of this as a programmer without requiring any new skills apart from the natural learning experience that one has to go through in any discipline. I think you can surely transition as long as your job requires it.