There is no point in that because the loss function itself is already an approximation. No one knows what is the exact loss function for any given non-trivial real-world task.
There is an optimal set of weights that minimizes the loss function for a given set of training data, but we cannot find it.
Granted, even if we could, it might just be overfitting.