So, I dove into this a bit (as a professor in a related discipline but who doesn't study weight) and had some heated conversations with researchers in this and related areas. My impression is that those studies are controversial because they "define away" diet effects.
They often engage in a bit of semantic game-playing about the meaning of the term "diet."
The problem is a lot of the "diets don't work" conclusions are based on very strict assumptions about "diet" meaning "telling people to just lower their calorie intake" or "telling people not to eat certain foods." They don't necessarily mean giving strategies for lifestyle changes, or ways to avoid increased calorie intake, or replacements.
As a concrete example, one researcher I spoke to is a very publicly prominent advocate that "diets don't work," and gave a talk I attended. She was discussing studies of weight loss and was explaining how studies that instructed participants to do things like use smaller plates (to change perspective on how much food is being consumed) or strategies for coping with parties, etc. were not "diet" studies because they did more than just tell participants to diet out of sheer willpower.
So, in effect, diets, if they are defined as just telling people to lower intake, and nothing else, don't work. However, if you support them, and give them alternative behaviors or foods, or strategies for lowering intake, they do work.
What also became clear to me, and which people on this forum are maybe pointing out, is that there's also a lot of gamesmanship in these papers about individual trends and what counts as a control group. There are in fact people who do reliably lose weight on diets (you can see it in their individual-level-data graphs), but their trends are often explained away by switching controls ad hoc.
I was left very frustrated by these studies, because it wasn't clear to me why these researchers wouldn't be promoting the strategies, etc. more, or advocating more for understanding why people who lose weight on diets do lose that weight, to gain strategies. I got the impression they were trying to be sensationalistic or something, to gain attention, by trying to sound counter-intuitive.
As a side note, I've worked in bariatric clinics, and I can say from my experiences there they have their own brand of burying side effects and defining outcomes in such a way as to achieve success (for example, by only selecting "appropriate" individuals for surgery).