Speaking as a physics professor, you can listen to me talk about awesome physics all you want, and maybe feel like you're getting something out of it. But if you actually want to learn math or physics at any level beyond "I've heard those words before and they sounded cool", you absolutely must be doing it: solving problems as practice, struggling with how to apply subtle concepts to new situations, and just absorbing the details for a while.

Are you genuinely envisioning this math/physics beach vacation as "mostly homework"? Because if it's not going to be a completely surface-level experience, it's going to be "mostly homework". :)