I'm also having trouble with the idea that the author of this trojan might be GCHQ or NSA (or sponsored by them), but that cryptanalytic techniques from the late 1970s are off the table.
I'm also having trouble with the idea that the author of this trojan might be GCHQ or NSA (or sponsored by them), but that cryptanalytic techniques from the late 1970s are off the table.
I'm saying that because the author is sophisticated, they know that the strength of the algorithm doesn't matter (so long as it's not something ridiculously simple).
DES, AES, RC5, RC6, whatever. They're all good enough because Symantec (or whoever is analyzing the malware) is going to get the plaintext no matter what. Symantec is not going to do any cryptanalysis whatsoever. Symantec is going to run the sample malware in a debugger and watch it get decrypted. The malware must decrypt itself in order to run.
This could be worse. It could have encrypted routines for which the commands are the decryption keys. It doesn't appear to be particularly polymorphic. If there's a BIOS/PCI dropper, it's (sadly) long gone here.
Look at Flame's encryption functions [1, §4]. This is the malware that burned a chosen-prefix MD5 collision elsewhere, but still went ahead and used the simplest mangling functions possible. You also find plenty of other 'state-sponsored' malware using similarly stupid/weak/outdated encryption.
They might as well XOR'ed the data, but opted in for RC5 for some reason.