When I was tutoring music theory, some students who had a similar experience had "aha" moments when we tried to focus more on how the harmony of a piece of music is linked to the meter.

So I wonder, how much do you hear harmonic changes in a given piece of music? Maybe a better way, for you, to develop an ear for rhythm is to focus on the regularity of those harmonic movements, which are what really "suggest" a meter in the first place. In most music, rhythm really is just the subdivisions of those larger "beats" implied by the harmony, and discussions like the OP end up being more or less "Oh, this is interesting because it doesn't line up with the expectations that accompany rest of the surrounding music"