I've been in the industry for 10+ years moving through the ranks in all kinds of setups, from developer to senior management, and I think you expressed nicely the gist of it:
The irony is that you don't really notice a great manager. Their job seems like a walk through the park, and you have a "I could do that job easily" feeling
Very similar to what you're saying is this quote from Laozi, a personal favorite:
A leader is best when people barely know that he exists, not so good when people obey and acclaim him, worst when they despise him. Fail to honor people, They fail to honor you. But of a good leader, who talks little, when his work is done, his aims fulfilled, they will all say, "We did this ourselves."
It's a tough path to walk on, as you have to let go of your ego, but seeing the team succeed always makes up for it.