> I've asked on HN a couple times in the past for citations that support this meme and haven't yet found a good source on this.
This is not a "meme", it is almost universally accepted science by nutrition scientists at this point. This includes both theoretical reasons, as well as actual, legit studies with human subjects.
> Calories seems to be the catchall metric pushed around, but it seems to be as much about what you eat, not just eating less.
Not sure what you mean by "it" when you say "it seems to be as much about what you eat."
> Whether calories is the thing that matters is the question.
Matters for what?
If you mean in the strict sense that you are able to control someone's total food intake, then yes, you can induce any amount of weight loss you want by limiting their caloric intake.
If you mean "matters" in terms of getting real people in the real world to actually lose weight? That's a much more complicated question, because the number one variable that controls the amount and duration of weight loss is how long you stick with the diet.
Any "diet" that causes a caloric deficit will cause weight loss. Some do this by directly counting calories and targeting a lower number. Some do this by causing you to eat foods that make you fuller, thereby making you limit the amount of calories without thinking about it. You can even take medication that just cause you to not be hungry, which is the most effective in some sense.
The thing is, most diets are hard to stick to, or at least most people don't stick to diets. I personally counted calories as I found it the "easiest" in some sense (though I also had to learn a lot about how to eat in a more satiating way so not to be hungry.) I used that to lose a lot of weight. I've now regained a lot of that weight, partly planned (as part of a bulk), partly unplanned (I got injured so stopped training, and then decided to focus less on weight control and eat more, knowing that I would gain weight in an unplanned fashion.)