It
does scale, and it
is that simple. I exercise regularly (run, lift), and if anything, I've only gained weight while on an exercise-only program.
On more than one occasion, I've made relatively mild cuts to my caloric intake (say, 250 Cal/day), and lost significant (>10 lbs) amounts of weight in 1-2 month time periods (and I've kept it off for years). Right now, I maintain my weight (6'2", 180lbs) by eating about 2,300 Calories per day. I don't count Calories anymore (I can estimate pretty well), but by weighing myself daily, I adjust what I eat to maintain.
People think that calorie restriction doesn't work, because they don't give it enough time, they aren't careful about it (i.e. sloppy record keeping), or they cheat. But if you're diligent, patient and honest, it's the most effective weight loss routine you can establish. I recommend using something like calorie-count.com to keep track of everything that you eat for a week or two, then cutting back on that number by a small amount (say, 250-500 Cal.) for a month. You'll be surprised at the results.