Of course people always had the feel-good lie "oh they're just being relocated to XYZ" but in those times you'd never leave your furniture and other valuables behind when moving if you were not forced to. For German people it was a win-win situation: More work for everyone (either as a party soldier or in the construction), steal some valuables from your neighbors who just got taken away, and feel good about your noble aryan genes.
Sorry for rambling on this topic but there are books for every mid-size Germany city which detail the unfathomable amount of looting, stealing and "M&A business" that was done by everyday "normal" German citizens during these times.
And most of these crimes were not prosecuted because of political decisions after the war.