I'm a big fan of Tesla and Musk, and you can see that from one of my recent comments on HN, too, but I've pretty much always criticized the Autopilot for exactly this reason.
Humans can't handle Autopilot in its current form, no matter how many asterisks Tesla puts in its Terms of Service and how many prompts and alerts it adds to warn the drivers. Either it's completely worry-free "let me take a nap until destination" self-driving - or it's not, and the driver should be expected to do manual driving at all times.