You are assuming that the average person is a weight lifter like yourself. Most people aren't, most people get very little structured exercise.
The truth of the matter is that most people don't have the time to go throw 45's around a gym for an hour a few times a week, and that diet and metabolism are the two main factors in a persons weight.
Edit: Also, most people aren't eating 3500 calories a day. They are probably eating maybe a couple hundred calories more than they should, but the calories that they are consuming is causing insulin resistance which causes your fat stores to take up additional calories. Even cutting out the extra calories won't help once insulin resistance happens. And here's the kicker, not everyone's body will exhibit insulin resistance, that is why there are people that seem like they can eat what ever they want and not get fat.
Here's my anecdotal evidence, I used to eat whatever I wanted, though I ate mostly (what I thought to be) healthy, eg: oatmeal, fruit, veggies, etc. But I hurt my back and all of a sudden, despite my reduced caloric intake, the pounds started being put on. It wasn't until I discover HFLC diets, and began to do my own research that I realized it wasn't how much I was eating, but what I was eating that was causing the weight gain.