Very few teams have the opportunity to switch later on, and if they do, it's at the expense of the job of the person who chose to go without a framework.
I estimate that every 20% that a codebase grows it needs some amount of refactoring. Definitely anything that moves from skunkworks to production will need some pretty heavy rewrites, and will probably need a framework that better meets its newer needs. The question is how much cost was sunk in getting there. The refactor will have to happen no matter what.