That's a failure of the argument, not a failure of philosophy. Everything from ethics to epistemology has philosophers arguing against unjustified assumptions. One could say the whole of philosophy is picking out unjustified assumptions. It's perfectly fine to start with a conclusion, so long as you can also argue your way there. If you don't think the argument is valid, then say so. What do you think philosophers do all day, just agree with eachother on every statement?
>is that mathematicians don't use their self referential axioms (in a situation which does not connect to the real world)
Yes, they do. Plenty of abstract mathematical concepts have zero connection to anything in nature or in the physical world. Some philosophers of mathematics even argue that mathematics has no root in the physical world. Besides that, the position you're arguing for is known in philosophy as 'pragmatism' - so don't pretend it's not philosophy. You're assuming your own axioms here.
>then claim that they are the experts on literally everything, because well "truth" and "ethics", by definition are everything.
That's not true; moral philosophers confine themselves to the world of moral philosophy. Epistemologists confine themselves to the world of epistemology. They don't claim knowledge about, say, physics or biology.
>a mathmatician isn't going to say that they are, by definition, the experts on truth and morals, and therefore I should do what they say anyway.
The definition of morality is literally 'what you should do'. If you disagree with that, then talk to a philosopher or just post an argument somewhere. If you don't think morality exists, then congratulations, there are philosophers who argue that too!
I'm not impressed that your argument rests upon the simple fact of calling their arguments circular (i) without specifying why and which arguments are circular in particular (ii) and saying that there are no goals to be acheived by even talking about, say, epistemology.
Nobody said you "have to listen" to anyone. You don't have to listen to scientists, mathematicians, logicians, historians, or anyone, really. But if faced with an argument you can't counter it ought to be to your embarrassment that you refuse the conclusion without considering the argument itself. All of your arguments, all of them quite philosophical in themselves(!) could equally apply to any other discipline with 'circular' axioms, such as physics (the principle of universal uniformity; the reliance on fallible observation and testimony) or logic (the axiom of non-contradiction).