You eat X calories, you use Y calories. If X > Y most of the difference is stored (as fat), if Y > X your body burn the energy stored (fat, at some point, but it depends).

It's simple math and it's scientifically proven. 2000 kcal are 2000kcal, it doesn't matter if they come from a single burger or 10kg of lettuce.

It's not what you eat, it is how much you eat.

Calories do matter. It just so happens that when we make bad choices of food we take in much larger amounts of calories. Sure you can say certain macronutrients are better for building muscle/losing fat but in the end a caloric deficit will cause you to lose weight. I'm eating much more volume of food right now but still losing weight. It's hard to pack on calories eating just chicken, brown rice, and veggies. I could get 1 big mac and take in more calories than my entire day of chicken/rice.

Some people can eat anything and naturally regulate their caloric intake.

Other people can eat whole foods (meat, vegetables, fruits, etc.) and keep a stable weight.

Others have an appetite that isn't well linked to their caloric intake and try to use external markers to guide what they should eat.

As long as it works, go for it.