When do you think it's too late though?

In some of the companies I've worked at, design was the last priority and at some point, making the effort to switch to a proper design system and do the front-end migration seemed too daunting. So it was never prioritized, until the exit.