That sounds like really bad insulation. Energy, in form of a hot or cold air shouldn't simply disappear from a properly insulated dwelling