You should never leave a sensitive device like your laptop unattended. A sophisticated attacker can replace your bootloader with a backdoored version that stores the encryption paraphrase to disk in plain.
That's what safe boot is for. You can't do this on a chrome book or a windows 8 laptop which is safe boot locked.