My dad has also always refused managerial responsibilities, but he was very much appreciated at his company. He said he had nobody below him and only one person above him, and he made more money than his boss (until his boss was replaced and the new boss gave himself a raise so he made more). My dad did get responsibility for all sorts of projects, just not for the people on those projects. And he has always remained very hands-on, sometimes replacing an entire team on his own.
It's basically what I want, but a lot of companies are apparently structured too rigidly to allow this.