>Why would an employer care whether I took time off to pursue other things otherwise?

I agree, and if they do?

I'm not telling you to lie on your resume, but...have a good story to tell about what you during that time off. If that doesn't cut it, you're probably not dealing with a company with a good culture.