As somebody who did all three, the difference to me is huge. I can do programming half awake no problem but math and physics require way more sustained concentration. Part of it is that with math, your compiler and debugger is your brain and it is faulty. With programming you offload a huge part of the mental strain to the computer.

When I'm programming, I often start tackling a complicated function (I do ML and numerics) without knowing the logic 100%. I just let my instinct guide me and I stumble little by little onto the solution with the help of the compiler. You can do the same process with math, but the mental strain required is orders of magnitudes more because you need to actually do the math yourself.