I suspect he means that he never stopped, hence nothing to go back to