The calories you consume affect your weight differently depending on your level of metabolic activity. So the key mediator here is how your body processes the calories you consume.
But talking about calories is wrong in any case. All calories are not the same. For example increasing glycemic index by consuming sugars and carbohydrates will lead to greater and more prolonged weight gain than consuming the same calorific load in protein - https://pubmed.ncbi.nlm.nih.gov/12081852/
It's simple misinformation to focus on calories rather than carbohydrate levels, physical activity, healthy gut bacteria etc.