I guess it would be one of these:

https://en.wikipedia.org/wiki/Astronomical_coordinate_system...

But I think for this study it doesn't really matter. As long as all the galaxies are in the same coordinate system, they can calculate the relative distances between any two galaxies. And since they all came from the same sky survey, all the data was probably already in the same coordinate system.

I think they avoided the problem by working with topology? But I barely understand this article.

right ascension and declination, i.e. equatorial coordinate system