This is called the Dial-a-ride problem or pickup and delivery problem with time windows in optimization literature in case anyone is interested.