You mean that by improving search we can solve any problem? What if solution field is infinite, even if we make search algo 10x100 more performant, solution field will still be infinite, no?
Gradient descent is a search. Where does it say the search space has to be small?