How exactly do you plan to do a PhD in theoretical physics with a background in CS? Sure, programming will come in handy and everything, but there is still physics to be actually done, especially in a PhD.
I've taken upper division physics classes but not enough to get a bachelors in physics
I don’t know your physics classes, but how often have professors said something along the lines of ‘that’s an open question’ in these classes? If that wasn’t the usual response to any sensible question in class, then those are likely insufficient to get you started with a PhD. YMMV, of course.