The problem arises when they're used to generate Database records, which is a common approach in Rails applications. Because you're generating a lot of them you end up putting a lot more load on the test database which slows down the whole test suite considerably.
If you use them to generate purely in memory objects, this problem goes away and then I also prefer to use factories (or generators, as you describe them).