Why I don't like factory_girl
blog.steveklabnik.com
blog.steveklabnik.com
The problem with factory_girl is that it lets you hide this pain instead of fix it, letting you get on with building massively coupled systems that turn into maintenance nightmares.
As a side note your unit tests should never hit the database or any external service. Your unit test suite should finish in at most a few seconds. If it's slower than that you have problems.