You can't do that at any team size. The role you was looking for is generally called a "lead" role, which has different responsibility than manager role.
It's mostly because of conflicting mindset and commitment. As manager, meeting is your tool to get work done (source: High output management by Andrew Groove, I didn't make this up, and I didn't understand it until after I had to become a manager either). As an IC, meeting is a big giant distraction to you. You basically can't get the two type of works to mix together. You will always be blocking other members of your team in your IC task. And this doesn't get started with the different mindset yet (For example, you have to watch/ teach people fixing ugly code, as opposed to doing it yourself. And there will be some point you have to give up asking polishing certain piece of code, due to the members not interested, incapable or other reasons).
Being a manager and part of the force should only be a transitional period.