Did you quit being a manager or the company? (Honest question). From my observations, it is rare to work for someone as a manager and not succumb to upper management pressure.
It was kinda awkward initially, mostly for the people I had hired and been manager of who I was now colleagues with, but not for long. Hands down it was one of the best life decisions I've ever made. I feel as though the experience I gained helps me to perform my current role even better than I would otherwise be able to as I have an appreciation of the politics above me that I'd not have had I not been up there and back down again the way I have.
I've recently had to step down from a senior non-management role (tech lead) that my company forced into management tasks into it, specially things that no one else wanted to do. It didn't work and I ended up leaving the company. It was hurtful because I left believing I had a lot to contribute and was motivated too.