I recently quit my job to run my own business, and I returned to programming for a living once again. Some observations on the differences between the 22 year old me in grad school and the 41 year old me as a programmer:
In terms of raw stamina to power though writing code, 22 year old me won hands down. I recall taking about 45-90 days to churn out a 35KLOC piece of Java code, One sprint during those 90 days was a 48-hour non-stop effort. I still have that code, and although it still works (with some changes) it's not the best code I've written. 41 year old me can barely manage 10 hours a day of coding before being too tired to continue. But I'm an order of magnitude more effective in getting the job done. And several orders of magnitude better at estimating how long a programming task will take. This happens because I've developed an intuition over the years for avoiding design and coding paths that are 'risky' -- no threads in my code, for example.
To address your point of perception of others that you are moving downwards, there certainly seems to be a perception of it. Early in my business endeavors, I'd pitch my services directly as a contract programmer, and I found that folks would be very reluctant to hire me - especially as the resume had several years of marketing and product management on it. I also suspect they may have inferred that I was expensive. Not true. But that was the perception.
I changed tactics a little later. I would pitch similar services as a firm (with me as founder), and sales process went swimmingly! Now the resume, with a decade of experience in management at large tech companies and the MBA, would open the door to the CEO/SVP level.
In addition, many of my early referrals came from folks in my network who knew me as a 22-year old and were in CEO/VP positions themselves, and were glad to refer me to their colleagues.
So, my advice to you, would be to not handicap yourself by competing on the same playing field as your younger competitors. Simply competing on pure programming prowess, even if you're good, would take away your strengths. Just change the game. Use the management experience you have to your advantage.
It's similar to how you'll hear people in academia refer to "industry." It's a popular thing to put down.
The typical reasons why a developer might not like management are because you're spending more time with people instead of code, you're looking at the product at a higher level rather than dealing with the details of coding, you have to deal with more politics, and you might not have the tools to be successful (i.e. current set of employees were hired before you, not enough budget, other business decisions aren't compatible with success, etc.)
It was a great learning experience though, because before I became a manager I thought that was like a natural progression, something that I had to do, something that I felt I couldn't shirk from, something where people recognized me for my skills, but in reality it's just a shit job, a dead end, unless you're one of the few who advances up the latter, because you know what, all of my managers and my manager's managers were all just relaying their manager's will down the pyramid.
When you are a software engineer you see things you do create value. You build things that work and do stuff. When you're a manager, I have no idea what I contributed. What did I do last week where I earned my pay? I have no idea.
Anyway, I'm glad I survived and am back to writing code. It involved getting a new job at a new place as soon as possible before I got too rusty. I was writing code at home all that time so that helped.
I was curious because I had heard for quite a number of people who had switched tracks from engineer to management and found it unsatisfying but they didn't go into as much detail as you did in your post.
I've been tacking towards "out of management" for a few years and it is not easy. Taking a "step down" is hard but only because of my own petty status-seeking and I've gotten past that. What I can't figure out is the salary problem. There are some developers who can get paid as well as I do, but not any in my industry where developers are a pure cost-center - I don't work in a software company. Changing industry with 7 year old tech skills (aside from open-source and side-projects) without taking a huge salary cut is impossible.
The appeal of being a manager to an engineer is the ability to allocate projects and work on the interesting stuff. Of course, very few managers get to actually do that-- it might be the 5 percent who have managerial titles but few or no reports, all of whom are pretty self-motivated.