There's an Iron Maiden song inspired by that show and features a voice clip from it. It's called The Prisoner.
But yes, Portmeirion had a rich history before McGoohan decided that it was perfect for his idea. Noël Coward wrote Blithe Spirit while staying in the village. George Bernard Shaw, H.G. Wells, Gregory Peck and Ingrid Bergman were all visitors. Beatles Paul McCartney, George Harrison and their manager, Brian Epstein, were also regulars in the village.