So that distance implies a higher volume of stuff that you're standing on top of, but the mass of the stuff is what counts for gravitational attraction.
Due to the "equatorial bulge" (actual term) you are further away from that heavy center, than at the equator. And due to the inverse square law, you lose intensity of that gravitational force very quickly, the further you are away.
Hence: the smallest gravitational force you get is on a mountain in the Andes near the equator.