Cucumber/Fit/high level tests aren't fundamentally less refactorable. They have a looser coupling to code than unit tests, so when the production code is refactored they are less affected than xUnit tests.
The problem most people make is that they are trying to cover too much with Cucumber. And they use it solely as an automation tool without any collaboration whatsoever. That doesn't work, but it's difficult to get that message across to people.
In order for Cucumber and Specification by Example to be used well it is essential to provide tooling that appeals to the most critical piece of the collaboration puzzle: The business analysts (or the people with the requirements).
Nobody has solved this problem yet, but I think it is a solvable problem. We're trying to solve it with Cucumber Pro: https://cucumber.pro
Let's see how that goes.