Which I acknowledged. You can decrease (or increase) your set point, but there is a limit to that. Thinking that everybody has the potential to be at BMI x simply does not match reality. The vast range of human physiology is stunning. Everybody is different. Some people need to carefully watch every bite they take and they still struggle with overweight, whereas others can eat "what they want" and they are underweight.
Totally neglecting any genetic component here is equally as false as neglecting the impact of factors like diet, exercise and environmental ones.