The final cyphertext of an encryption system should be indistinguishable from noise - the less random the output looks, the more information an attacker gains.
The output of your 512-bit encryption should therefore look like noise already. Converting noise to noise should therefore result, predictably, in more noise. So I'm going to say "no, you can't tell".
Instead of auditing the code/file format, it may be less error prone to just re-encrypt the entire file thereby hiding any metadata that the encryption program attaches.
And that is before you get into less than perfect encryption.
Proper encryption, maximum compression, and random bytes are all indistinguishable, as they have maximum entropy.
I only failed 9th grade three times, so feel free to correct me.
without additional information!
I can prove to you that you're seeing an encrypted message by giving you the decryption key, and I can prove to you that you're seeing compression by giving you the decompression algorithm.
However, there is no way to prove that random bytes are _really_ random.
well of course, because the opposite is true, any string of bytes can by XOR'd to something meaningful.