Writing tests that are not highly coupled to implementation is hard. That means that every time you change the system, you have one or many test changes to make too.
How do you know what to test and what not to test? How many tests do you write? One line of code can, if you think about it exhaustively, require 3 or even more tests.
A lot of time you'll write code that facilitates testing. Do you test this code? What happens if the code silently fails and now you don't have proper test coverage of all the code paths said code is supporting? This is particularly true if you are mocking objects.
Learning how to test, properly, is almost as hard as learning how to code. My advice is, if what you're doing is boring, like most Rails apps, lean on the library as much as possible, break up anything that goes against the grain into unitary features, and test them each in turn. You don't need to test that ActiveRecord.belongs_to works.
It doesn't surprise me that there's developer reticence towards testing. I tried powering through that reticence, believing there's some "magical world" as you put it, where testing actually helps you code faster and more accurately, but I found even myself a TDD apostate, I don't even do it for my own projects anymore.