Another way to explain this is that quantum computing would be the perfect answer to some extremely-naive kinds of cryptography, reducing the decryption time-complexity quite harshly (though not all the way to O(N) as one might expect)—but modern, practical cryptography does things fairly differently than spherical-cow cryptography.
For example, if cryptography basically consisted of taking a set of plaintext "blocks" and a key, permuting the key separately and deterministically for each block in O(1) time, expanding the key into a one-time pad again in O(1) time, and XORing each block with its respective pad—then you could use a quantum computer to quickly search the keyspace for a key that decrypts the blocks into something sensible according to some heuristic. You can make a quantum algorithm that "searches" a static keyspace, given that you can map a particular ciphertext block to particular plaintext block in O(1) time—this mapping effectively becomes a "projection" of the keyspace, and the quantum computer searches that.
The problem with this approach is that, in reality, subkeys in a "stream" aren't generated independently per-block (as happens in the much-derided "ECB mode" of block ciphers), but rather are generated serially by feeding in the previous subkey in a chained operation; and that each cipherblock is then created by "expanding" the subkey through a CSPRNG, which itself uses iterated hashing.
You can't make a quantum algorithm that searches a keyspace, given an encryption algorithm that is defined recursively or "statefully" on its input stream. There's no such thing as a "quantum for-loop" that magically makes an O(N)-step process into a single linear projection of the keyspace—and without this, there's nothing to usefully search through.