This has been solved in the last year or so. Look up "LLM chain of thought".