Somewhat related: The size of a chain will be constantly growing, right? How fast, and will it be a problem?
If you have more than one currency installed on your computer you might use 100GB or more for them. It fits, but it's starting to become a significant % of a typical hard disk. (Although presumably people with lots of wallets are not typical and have larger disks.)
PNY has a 32gb usb key for $13 (about 0.029btc).
An offline live-CD or USB key has no need to store the entire blockchain if you just want to store coins and keep them safe.
(I'm aware of light clients that work with a server, but that's not the standard.)
You can store the wallet cold on a usb key, certainly, but that's not the use case here. The use case here is a usb-key you boot ONLY for bitcoin, and nothing else. No web browser, no nothing, just the bitcoin client.
This is recommended if you are on windows, or if you aren't certain you can secure your computer (most people can't).
Whether someone has coded such a client I don't know, but it's not a difficult problem, and does not require a server.
There are really two parts to the Blockchain:
1. Header, which includes the hash of the contents and other metadata like the previous node in the chain, etc.
2. Contents, which is what the miner chose to validate when the block was mined. In Bitcoin, this includes a set of transactions (a step from a graph of inputs and outputs).
The contents could potentially be pruned by only keeping track of unspent transaction outputs, but this removes the ability to validating the headers (ie. hashing all the transactions to check if the header metadata matches) except for the fact that there are other headers on top of it.
The set of blockchain headers will continue to grow, unless there is a new genesis block—this is like a forced snapshot of the current state of the relevant content of the network. In Bitcoin, it would include all the current unspent transaction inputs.
There are several blockchain technologies like Mastercoin which attempt to completely decouple the notion of a content specification from the blockchain headers themselves. That means you could conceivably send all kinds of garbage that would get happily signed by the miner but the contents would be ignored by any client that is not interested in it.
(Disclaimer: This is written off the top of my head and I gotta run, so it may be somewhat inaccurate.)
That said, as a reference, a Bitcoin transaction today is anywhere between ~160~1000 bytes, and the full block chain is on the order of tens of GBs. Is that a problem? It can be. To address this, Bitcoin uses Merkle trees and "Simple Payment Verification" (SPV) -- worth looking into, if you're interested.. long discussion on its own. I'll just note that SPV changes what you can say about integrity/security of the transaction.
Bitcoin has to become much, much more successful (think hundreds of millions of users) for the blockchain size to ever become a problem.