One thing I like to ask myself is how did things get to be the way they are?
>all the people who have dedicated their time to optimizing . . . over the years have missed some enormous low-hanging fruit.
When I was a teenager one of my most worthwhile early chemical breakthroughs as an undergraduate was a typical lesson (these are dubbed "experiments") where students produced a product from raw materials and were evaluated on their final yield of acceptable recovered product, compared to the theoretical amount that could be obtained if perfection were to be achieved across all steps in the reaction and subsequent processing & handling.
This was a reaction that was known not to go very far to completion when equilibrium was reached to begin with, so most emphasis was placed on technique for subsequent separation and purification of the desired product recovered afterward. Good yield was considered 25%, poor at 10% or less, the historical high score was 27%. It was also accepted that the final product was not considered very stable, subject to degradation from exposure to things like heat, air and moisture, and there were some literature references this was based on.
Basically an overall view of the combined laboratory techniques applied across the study, compared to how the other students were doing. The product was not actually a commercially useful material but it had proven worthwhile in this regard.
The university had been doing this same challenging competition for decades, designed back then by a still-active professor, and it was considered a good comparison of how each year's students were performing on the same real-world problems over the decades. Same old same old.
I went through it one time and it came out pretty good, but before refining my technique I dived a bit deeper into the physical properties of all chemicals involved, in my case looking for a way to drive the equilibrium further to begin with.
One of the well-known ways to drive reversible reactions to completion is to remove product as you go along, not always easy but also quite essential in many industrial processes.
Seemed to me distillation would be most feasible except the product was a solid and one of the raw materials was a liquid having a known boiling point, much lower than the expected boiling point of the dissolved solid raw material product in the reaction flask, and the solvent much lower in boiling point than that.
Ended up vacuum distilling from the incompleted reaction flask where the water of reaction was removed along with the solvent, "excessive" heat actually helped complete the reaction before the remaining lower-boiling raw material was vaporized, and the desired solid product had a lower boiling point under vacuum while melted than the remaining "heavy solid" raw material, and the good stuff was recovered in over 50% yield as an oil in it's own dedicated receiver, which crystallized wonderfully by itself with no further purification needed. Earned me my first A+ and encouraged me to continue going further than average ever since.
Turns out all the original professors were wrong about heat instability, and also had never fully considered as many physical properties of the exact chemicals being worked with, only similar materials for which there was much more common knowledge and published references.
The final lesson was that sometimes the most respected elements of "common sense" amount to more or less "common lack of sense".
Also, I've said this before, when all recommended solutions have been attempted and failed to deliver, the actual solution will be something that is not recommended.