If you can't fire or promote them are you really their manager?
In smaller orgs with no HR, it's probably easier, but larger companies have a specific process.
Though to your point, I would not consider myself someones manager unless I had the option to hire, fire, or promote them.