I love ruby having used it since before Rails, but it seems weirdly susceptible to the "with great power comes great responsibility" issue. Dynamically typed + the "don't disallow stupid things or you'll disallow clever things" mentality, maybe? Early adopters like _why who could actually handle it having a huge influence? Dunno.
I've worked with some devs who were not old battle-scarred greybeards who jumped on EVERY rspec feature because they were there and in a misguided attempt to DRY their test code well beyond reasonable limits.
So I feel you here.
But FactoryBot does allow fixture-like speeds with in-memory .build()'ing. Too few people use it, and too few have succumbed to a preponderance of tests being full-on integration style.
Ruby (and Rails, and rspec) all IMO require a lot more good guidance and wisdom on what NOT to use than other ecosystems I've used.