* You are responsible for sticking to commitments that depend almost completely on other people sticking to theirs.
* You're still on the hook for delivery, so if someone you assigned a subtask can't get that job done, you either have to pass it on to someone else (which often ends up harming the social dynamic of the team) or just do it yourself. Throwing an employee under the bus is not a good look.
* You get to be the "executioner" when a RIF rolls through, even if you get RIFed yourself, and even if you had no input into who is getting RIFed. Those Sunday evening "let's have a chat tomorrow morning" emails, and the following chat, are the most gut-wrenching things I've ever had to do as a manager.
* By the time you become a manager, you're probably one of the most experienced members of the team, so you are still always the "go-to" person when someone needs a gnarly bug fix, or if there's a lore question, or if a sticky situation with a customer comes up. If your reports are too green, it's hard to delegate these things, and this just adds to your own cognitive load.
* You don't have enough clout to make organizational-level changes. So process things that you see that are really inefficient, and you have some ideas how to address them? Now you've got to convince your manager and probably theirs, too, and they usually don't want to rock the boat.
Your best path as a ground-level manager is to not spend too much time here: become a second-tier manager (ie, director) or find a nice landing place as an individual contributor again.