Refactoring as a "yellow flag" is not only dangerous, it's the opposite of how you should treat it. It's not just about cleaning up code that might not meet the bar on quality. Sometimes it's about challenging assumptions by the developer that were completely valid when they wrote the original code that aren't now through hindsight, technology changes, etc.

It's possible for a developer to write all-star top shelf code only for it to be significantly improved a year later with a few line changes. To not challenge all code, even the good stuff, is doing your codebase a disservice.