In this case, there's multiple issues going on here. First 'gorm' in the case of 'fear gorm' didn't mean blue. It meant 'dusky', 'swarthy'. Also, 'fear dubh' refers to a black-haired person; 'An Fear Dubh' refers to the devil. 'Fear bán' would likely refer to
hair color. Irish traditionally classified people on their hair color (thus 'Roisín dubh' was 'Roisín of the dark hair', not that she was dark complected).
Another big issue is Irish colors don't match up to English ones, at all. So many seem to think 'gorm' matches up to English 'blue', when it doesn't. It's different. Same with 'glas' and 'green', which is why you can have 'capall glas' or a 'green horse', but really it's a different color split that English. Same with 'spéir glas' to describe a 'grey sky'. It's a huge problem among learners who learn the language incorrectly (most in Ireland, sadly).