It definitely requires ccie level knowledge and at least 10-15 years experience, plus advanced Linux/BSD server admin skills to really do five nines right. It is indeed expensive and requires enthusiastic cooperation from non technical management responsible for budgets.
I worked my way up from a level 1 NOC type position, so I'd like to think that after 20 years I have a good understanding of all the possible OSI layer 1 failure modes (and things you can fuck up in configuration at layers 2 and 3), yet the things some other partner and competitor regional ISPs do continue to surprise me.