True to the extent that it is actually related to your job in a larger sense and doing so won't adversely affect your ability to do your 'real' job. Many managers will just keeping adding more and more responsibilities to their 'best' workers until they either start saying "no", quit or burn out, often without any malice or ill intent.
That being said there are better ways of phrasing it than "this isn't my job". Saying something along the lines of "if I do that I won't have time to finish X by next Tuesday as I promised. It might be worth checking with Dave and his team if they have someone who can work on it. Otherwise I'll be happy to do it, as long as you're OK with pushing X back a week".