See my other comments about static profiling of kernels. There are ways of improving the search that keep runtime at the heart of it.
mcts / rl isn't really a heuristic. but yes heuristics can be used temporarily to keep the search space small, and removed over time as the search algorithm improves.