https://en.wikipedia.org/wiki/Demarcation_problem
Some examples:
1. C. S. Pierce believed he had discovered a third form of thought, distinct from deduction (math and logic) and induction (empirical science). He called it "abduction."
https://en.wikipedia.org/wiki/Abductive_reasoning
2. Certain aspects of philosophy seem to be outside what we consider math or science, such as ethics. The "is-ought" problem is currently open; we basically have no way of giving force to normative statements (you ought/should do a thing, it is right/good/just to do a thing). Kant's categorical imperative is probably the closest approach, but it's hard to accept it uncritically (or seriously entertain Kant's resulting system of ethics, which for example imply you should not lie even to save a life.)
https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem
3. Semantics (study of meaning) also may not be neither math (which can avoid all semantics and adopt a formalist POV) or science (which can also be described as a formal attempt to predict empirical observations, i.e. operationalism and instrumentalism.) For example, the logical positivists attempted to define a verifiability criterion of meaning, but this can easily be show to fail its own test, leaving the question open.