"Refined" is actually only a heuristic in regards to carbs, not an all by itself meaningful distinguishing feature. Refined carbs that are bad are bad for reasons having little to do with being refined; there are also thoroughly refined foods that are harmless.

Potato in particular is bad because it is a concentrated source of very high GI starch,