There's nothing wrong with sugar when eaten as part of a meal, it's when people down Coca Cola on an empty stomach that there's a problem. And fat is actually essential to life.

At the risk of stating the obvious, when you eat a meal, all of it goes to the same place. So as long as the meal as a whole provides sufficient nutrition (and if you are eating a varied diet then it absolutely will do), no part of it can be described as "empty calories".