The ability to reason about it or even measure it might be subjective, in the sense that most developers aren't really good at articulating quality. Also, developers often confuse familiarity and quality, or to be a bit more generous: are unable to accurately assess quality of unfamiliar technology.
Furthermore, there is also a widespread misunderstanding that if you can't 100% prove something beyond a shred of doubt, that any position is as arbitrary as a favorite color and any reasoning a waste of time.
> Doesn't matter what the code looks like, as long as the customers are happy.
Customers being happy or not is exactly the problem of Code Quality. When a system breaks down over time due to a pervasive neglect of quality, the customers won't be happy. In practice this is happening slowly over time. Your lead time increases, there will be more bugs and changes will cause more problems that are increasingly hard to overcome. It can also come as a collapse, for example a huge outage, data loss or security incident. This can kill your business.
Quality is not something done for developers sake, it is about the ability of making customer happy over time rather than just now. That is the ultimate justification, which isn't subjective, and the source of legitimacy of any time spent on refactoring at all.
If it is not ultimately rooted in solving a business problem, then it is not a quality improvement at all. And yes, sometimes developers try to masquerade personal taste as quality (refactor to something I know or like), and in these cases the outcome often isn't an improvement at all. In fact, it is often a regression. But that doesn't mean quality improvements aren't real or beneficial.