"The standard image of Southern slavery is that of a large plantation with hundreds of slaves. In fact, such situations were rare. Fully 3/4 of Southern whites did not even own slaves; of those who did, 88% owned twenty or fewer. Whites who did not own slaves were primarily yeoman farmers. Practically speaking, the institution of slavery did not help these people. And yet most non-slaveholding white Southerners identified with and defended the institution of slavery. Though many resented the wealth and power of the large slaveholders, they aspired to own slaves themselves and to join the priviledged ranks. In addition, slavery gave the farmers a group of people to feel superior to. They may have been poor, but they were not slaves, and they were not black. They gained a sense of power simply by being white."
http://www.pbs.org/wgbh/aia/part4/4p2956.html