There is not a black hole at your center of mass. Protons and neutrons (let alone atoms) are not dense enough to form a black hole. In all likelihood, your center of mass is empty space between a nucleus and some electrons.
In other words, if you are outside a spherical mass the pull of gravity gets stronger as you get closer (~1/r^2). But once you are inside the mass, the pull gets weaker. Hence there is no black hole at the center of the earth, for example.