It was suggested that I could take them to small claims, but I've never done anything like that before, and looking into it seems that Amazon makes it rather hard and admin heavy to do.
I've basically written off KDP for all future publications, and most of my current stuff is for sale through all the other book retailers.
It's a real shame as Amazon is the heart of a lot of indie publishing and provides a good service but ultimately they don’t care about the individual.
I have seen stories of slightly bigger indie authors also raising hell on Twitter and getting various other people involved (BSFA in this case) to speak to Amazon. In that case I believe they got paid.
I’m still shocked that Amazon let me pay for ads for a novel that I would never get the sales money for. That hurt more than just being delisted.