Very similar experience here! 4K+ calorie daily average mostly of junk food from ages ~15-21, maybe 30 minutes/day average of weightlifting, rarely some cardio, visible upper ab muscles, call it a "4 pack", full on six-pack abs if I happened to be slightly more active in a week (a few hours of casual swimming, for instance).
Then I decided I wanted to cut down to the six-pack bodyfat level and stay there, so for the initial "cut" phase of that plan I dieted at ~1400 calories a day, every single day (no cheating) for ~2-3 months, and started doing way more cardio. Only lost a couple pounds, little or no visible difference, so, confused and frustrated, I gave up. Makes no damn sense according to any "weight loss calculator" I've ever seen (which all claim that my BMR should sit in the low 2000s), but that's what happened.
Packed on fat immediately when I went back to my usual eating patterns, took me years to get that under control. To this day I seem to easily hit a plateau a bit over what I'd call my healthy range when dieting at any reasonable (not near or under starvation level) caloric intake.
OTOH I've found that strength training's more effective for me than it "ought" to be but requires time, which sucks. I'd love to be able to diet my way to a trim waist, since maintaining an intake under what weight loss tables/calculators say should work for me is something I find easy to do. Too bad it doesn't.
Don't know what the explanation is, but something weirder than the "calories in/calories out, and calories out is fairly easy to guesstimate with useful accurately" orthodoxy is going on.