This is possibly my favorite PG essay, the metaphor about gliding seems very relevant to your question: http://www.paulgraham.com/hs.html

My two cents: If you’re interested in engineering and physics, you should explore that. Does your university have a CompE or EE program? At mine a lot of the classes between different eng programs overlapped so it wasn’t difficult to switch.

But you could also just pivot because no one in the tech industry really cares if you have a CS degree, and it sounds like you’d still enjoy coding anyway so you’ll keep learning on your own. The CS curriculum is going to feel painfully old fashioned compared to what you’re learning on your own, though there are some useful fundamentals in there.