First, can we agree that any encryption algorithm better than a Caesar cipher is good enough for this particular application? That's because nobody who's analyzing this malware is going to bother with cryptanalysis because there's an easier approach. The encrypted data has to get decrypted before it runs. Therefore, you watch the execution with a debugger until it gets decrypted and executed, and then just capture the plaintext data (and the key if you want).
Given that resisting serious cryptanalysis is off the table, I can suggest several reasons for RC5:
- It has much smaller code size (500 bytes vs 15000 for AES).
- It's less vulnerable to fingerprinting. That is, anti-virus software might have checks for AES but not RC5. The malware writer would know this because they actually tested their malware against all the anti-virus products and found that none of them have signature or heuristic checks for RC5 but do against AES. (I'm just fleshing out the thought; I don't know if AV software checks for AES and RC5.)
- AES runs faster than RC5 in general, but maybe the particular platform or instruction set for this malware makes RC5 faster.