The point is that it requires space. We have to keep a record of the pointer.
Why do everybody needs to keep everything? The chain will be much smaller with just the id and the hash of the block. The payloads could be kept only by the interested parties and could be validated when needed with the chain.
Because every node in the network must be able to verify all transactions all the way from the genesis block. In order to ensure that this is possible, everyone who keeps the history must keep the full history.
[1] http://mimblewimble.cash/20160719-OriginalWhitePaper.txt