Some of those celestial bodies are even small enough that we could build the elevators with existing materials, too.
The problem is: the benefit of a space elevator is bigger the deeper your gravity well is, but it's also much harder to impossible to make one there.
But I've also been wondering: wouldn't it be possible to have a tapering space elevator? Reduce the weight by making the parts that have to carry less weight thinner?
A steel cable would be 1mm thick at the bottom and several light years* thick at the top.
*hyperbole. Maybe.
A lightyear is around 10^19mm. Given that it's by area, a square lightyear is ~10^38mm^2. The cable is about 100 times thinner than it.