If he's pulls over and asks you where you're going, then he's not actually off-duty. If he refuses to take you (either because you don't answer or because he doesn't like your answer) then you should report him.
Source: http://nycitycab.com/NYC%20Taxi%20Guide.aspx#offduty