The slave who manages an important man's affairs is himself an important man.
Concubines frequently wield power through their well-placed lovers. At the historical extreme, Wu Zetian was a concubine who worked her way up to being formally crowned emperor.
The Mamluk Turks were slaves. This didn't stop them from taking control of the government and even enacting a requirement that the Sultan be a Mamluk (children of Mamluks, not being slaves, were not eligible).
Chinese eunuchs were also slaves, and also took control of the government on multiple occasions.