All recommendations are for Bitcoin.
Most secure: A hardware wallet like the "Trezor" or "Ledger". Good balance of security and usability. Gives you a backup phrase in case the device is lost. Backup phrase fits on a piece of paper, can be made as secure as required. Keep backup phrase in multiple locations.
Cheap and secure: (encrypted) paper wallets generated from https://www.bitaddress.org , with backups stored in multiple locations. Bad usability, but can be made as secure as you want/require.
Easy, reasonably secure, not for large amounts: "breadwallet" for iOS/Android. OSS, user friendly, but not recommended for long-term storage. As most modern wallets do, this one also gives you a backup phrase to be stored on paper.
As for exchanges, I'm in Europe and trust bitcoin.de. On Reddit there have generally been no major complaints about Bitstamp, a few about Coinbase, and a lot about Kraken.