I thought it was pretty clearly defined: they product offspring which can reproduce and produce viable offspring among themselves but not with the original population
A can mate with B, B can mate with C, but A can't mate with C.
The question of defining "species" was one of the keystone questions used as an example in a philosophy of science course I took. See: http://plato.stanford.edu/entries/species/