I think you do since not having it means that your wallet wouldn't know if you've spent or received any coins.
Here's a quick example: https://www.vox.com/2017/6/20/15839626/disparity-between-urb...
Edit: If you give someone a coin with the private key on paper, technically you can give them coin offline, but the rest of the networks have no clue this happened, and you can't spend unconfirmed monies.