They wouldn't need to be home. The self-driving car would drive to the customer.

True, but given a choice between an amazing customer experience and saving money, most people choose the latter. See: ATM, phone tree, commercial air travel.

I can just imagine the grandmother in a walking chair trying to get the package in a timely manner.