As someone whose had to carry around five figure sums around quite a bit in the past because of my profession (hint: it rhymes with "rambling") my far and away, it's not even close, fear is having the money taken by cops from an illegal search. This fear is justified because I've never been mugged or robbed and no one I know who was in the same line of work has either. But I know MANY people who have had funds seized by cops. ESPECIALLY in the south. Actually the horror stories I personally know about have happened only in the south.
Usually it's the cop seeing an out of state license. Pull you over, lies and says you were "swerving" or refuses to give you a reason. Then forces you out of the car, takes the money and sayonara.