Crime is an issue but shouldn't be a complete deterrent. The political environment is a bigger concern scares people off. The BEE/indiginization issues are very real and should be considered.

With that said, I would live in Cape Town in a heartbeat, great food, culture, people etc.

Not really an issue if you live in the right areas, in my experience. Cape Town in particular is quite safe as long as you don't go into the townships (like Rio's favelas).