I'm not seeing much of an argument against factory_girl (in unit tests) other than: it's not as fast as setting up your model by hand. And no description of the alternative?
The problem with factory_girl is that it lets you hide this pain instead of fix it, letting you get on with building massively coupled systems that turn into maintenance nightmares.
As a side note your unit tests should never hit the database or any external service. Your unit test suite should finish in at most a few seconds. If it's slower than that you have problems.