The security concerns start happening after an address spends a UTXO. Before a P2WPKH (segwit) address is used, only the public key hash is known. In order to spend from it, the full public key needs to be revealed. That's why it's recommended to use single-use addresses, because a quantum computing attack or elliptic curve vulnerability could be used against an address where the attacker knows the public key, but would not work against an address where the pubkey has not yet been revealed.
So, the main security change happens after you spend from an address the first time. Subsequently, there are theoretical vulnerabilities that could occur after an address is spent from many times, but really only if the signer is malicious like dark skippy, or faulty and doesn't properly follow RFC 6979 deterministic signatures, leaking some signature entropy which could be used to crack the private key. The latter has happened with some bad custom wallet implementations, but these attacks are even further in the realm of theoretical, not super realistic, require faulty software/firmware to be implanted into signing devices.