I would highly recommend giving this excelled LessWrong post a read: https://www.lesswrong.com/posts/rarcxjGp47dcHftCP/your-llm-a...
It isn't a perfect fit, since the article talks a lot about the scientific method which doesn't apply super well to philosophy+math, but I think there are some strong parallels here.