It's actually somewhat more complicated than that even: clicking an "empty" custom colour (I put "empty" in quotes because it actually has no concept of empty, they're just white -- it keeps track of how many colours you've added since opening the dialog but it does not know if the colors you're overwriting are "empty" or not) selects the starting point where you want to start overwriting the list. So clicking multiple times will keep overwriting your colors, you just get to choose your starting point. So there is kind of a way to use it but it is non-intuitive and backwards as all heck.
What the "add" button does, then:
1. overwrite colors step by step in the list (with no undo),
2. in order from top to bottom and then left to right,
3. starting at the most recently clicked custom color,
4. or at the first color in the list if no custom color has been clicked since opening the dialog
Easy! :)