I will do a double blind experiment where I do startups in one version of my life and don't do them in another. Then I'll know whether doing a startup damaged my career.
Ok, just kidding.
I knew it was the right move when I realized I was learning 5 times as much in the startup.
Nothing matters more to your career than the development of your brain.
And if you're not excited by what you're learning, if its become stale and dreary, make a change.