Allowing your application logic to do this for you creates all sorts of opportunity for hard to debug issues that have nothing to do with the thing you were actually testing. Putting data into the test DB directly makes for more setup, but makes for much more isolated test cases and much more certainty that your tests are doing what you think they're doing.
It also means you can write tests as you go without having to build a whole level of application logic to support your single test.
It is not a chore to unit test code, it is an extremely useful development tool that forces you to check your assumptions and verify your design as you go to give you a better product.
The attitude of this comment is along the lines of the sorts of attitudes towards testing it takes a lot of effort to un-learn junior engineers of.
However, everything has a cost, even abstractions. The more powerful the abstraction, the higher the cost. I find that for most of our projects, the sorts of abstractions unit testing forces on us is much more powerful - and thus higher cost - than necessary for the task at hand. If you are not taking advantage of that power, then all you really have left is the high cost.
Then some poor fellow ends up redesigning the test suite to actually follow the assumptions of a unit test's test data being only ever accessed by that unit test. I have been that fellow, and it can be, bluntly, quite a pain in the ass; as you're doing it, you can't help but think, "whoever wrote this was too lazy to set up the test data properly, I shouldn't have to spend the time to fix this now."
expect(Klass.new.special_case).to have_this_behavior
Anything more complicated in test code should be considered an anti-pattern.