So we can let the ai generate some math problems based on the solutions found? Other fields (computer science, physics, ...) can generate math problems too.
They're good to give to new mathematicians, and they're good to help humans understand the shape of the problem space and relative difficulty with the tools we have.
Cheesing these problems with LLMs gets rid of both the training benefit and our ability to create good related problems. There's an aesthetic part of this, too, that LLMs do not capture.
But from paleontologists' perspective, they go out looking for dinosaur fossils when they have questions that digging up a fossil may answer. The metric they're focusing on isn't tons of fossil mined out of the ground, it's a developing understanding of extinct life.