20 years ago my first job out of college was with a small company doing combinatorial auctions for the institutional bond market (among other industries). In the end the chicken/egg problem of having enough liquidity was too much to overcome.
At one point we pitched the NYSE on doing the opening call as a combinatorial auction but they were not interested.
We did have some success in pollution credit and trucking logistics markets though.
Of all the jobs I've had, that was easily the most fun/interesting algorithmically. I got to learn all about LP/MIP solvers, graph decomposition, distributed computation, etc.
Best of luck to you!