You don't need >50% of bad actors to compromise the blockchain, but rather >50% of the total hashing power. This could very well be achievable by a small group of people with QC at some point.