What tools would work best / support that workflow?
In your case, all of the full libraries like proj4 already support it. What you're describing is a custom ellipsoid and datum. Plug that in, and you're good to go. You'll need to define the shape of your asteroid's equipotential surface through a reference ellipsoid dimensions and a datum that defines deviations from that ellipsoid. Making those is definitely non-trivial, but once you've got the datum grids made, the math is the same as it is for Earth.
Also note that the datum is _not_ the shape of the asteroid directly. It's the 3D shape of a surface that's "flat" gravitationally. In other words, it's the shape of sea level. It's something that depends on the density distribution of the body and it's really not easy to measure, though it is easy to predict if you know the density distribution a-priori.
The shape of the asteroid's surface is topography. That's independent of the coordinate system.
Mapping frameworks need to support map APIs to render stuff, and some APIs such as Open Geospatial Consortium's WMS (OGC WMS) requires clients to specify coordinate systems.
Yes projections need to be specified... The datum and ellipsoid are part of the projection. No, you won't be able to pass your custom datum along to an external service and have it recognize it automatically. That doesn't change the fact that you can define it locally and have your local tools recognize it. proj4 et al will happily let you define custom ellipsoids and register a new datum and work with projections on that ellipsoid and datum. You can view that just fine in QGIS et al.
Also, WMS is pretty niche and not too widely used partly because it supports things like this use case and is therefore quite complex. I wouldn't bring WMS into this. However, if you want to, the projection specifications it uses _also allow for custom ellipsoids_. You can happily request things in a projection with a non Earth ellipsoid -- it's just a matter of whether the WMS server supports it.
I would have stated this first.
You don't really need a equipotential surface or even vertical datum, unless you're trying to model potential energy, uphill and downhill, the flow of a river, or the gravity well. All of which are unlikely to be known or easily measurable on an astroid.
And if all you want to do is look at the geometry of a surface, you don't need, or even want to use a horizontal datum and projection. It will just overcomplicate the math.
Lat long or a similar polar (or elipsiodal) coordinate system using orthometric height from the origin (or an arbitrary "zero" datum height) will be much easier. Especially in the case of a body without existing coordinate systems or projections that need to be "tied-in."
On the other hand, there are cases where using the GPS style coordinate system makes more sense (which is just an x,y,z or x,y,H version of lat, long, height. Yes, at it's heart, GPS uses Cartesian x,y,z). This is especially true if you're not concerned with defining an ellipsoid, prime meridian or equator. (Especially true for shapes that might not rotate, or fit well to an elipsiod or sphere.)
The thing to remember is that coordinate systems and projections mainly exist to translate between the many different ways people have been mapping and measuring the Earth for hundreds or thousands of years. Sometimes there are good reasons for these different methods (shape, area, directional constraints), and sometimes it's advanced tech/ systems (SI vs Imperial, Clarke elipsiod vs Hayford , or convenience/locality (relating Rose line observations with those made in Greenwich), or national security (USA not wanting nuclear icbm targeting stolen by USSR to be useful/meaningful/etc).
I say that to say this: If you're doing everything from scratch, there's no need to complicate things. Stick to the simplest math that works for the projects demands. There's nothing wrong with spherical polar or xyz.
My point was more that the standard mapping tools allow non-Earth bodies just fine. E.g. here's a UTM-like projection for Mars using a simple ellipsoid in proj4: "+proj=tmerc +lat_0=0 +lon_0=0 +k=0.9996 +x_0=0 +y_0=0 +a=3396190 +b=3376200 +units=m +no_defs" You can happily define a simple sphere of the right general size and have things work correctly.
It's not overcomplicating the math to use existing libraries, it's avoiding rewriting a lot of fairly complex operations that are easy to get wrong.
You can use gdal, proj, org, et al quite well on other planets. It's a hell of a lot easier than writing everything from scratch. They're set up to do things like this.
Although admittedly I haven't kept up with the different capabilities for the last few years.
What method/format would even be used to generate points for an astroid? Radar? Lidar? Photogrammetry? From a "where do you start" perspective, knowing that would probably narrow down practical approaches...