I ended up using gallon sized ziplock bags. For each cable type/connection, I put them together in a bag. Then I have all those bags labeled and in some little bins on a shelf that fits the bins. When I need something, I can slide out the bin and flip through the bags like file folders, until I find the one I need.

It's worked decently well so far.

For a long time I hand a giant 40 gallon garbage bag full of a rats nest of cables. Then I went through that to get rid of old stuff I didn't think I'd ever need again, and ended up with a rats nest in a box. Now I have them all sorted.

It's probably not ideal, but it does a decent job. It is a bit more space efficent than the plastic tray/tote my company used in the data centers to sort all the various cables. That took up a whole wall.