The point of refactoring is to make the code cleaner and easier to understand.
My thinking is definitely more focused on early stage start ups/personal side projects than bigger mature projects.
But what I'm trying to get across is that I'm starting to wonder if "big clunky code base that's hard to support, maintain and understand" is in the same class of good to have problems for early stage start ups/side projects as "I chose a technology that is tough to scale to 500 million users" as long as you have a comprehensive test suite.