On the off chance you see this... I don't want to get too close to describing things as I'm trying to keep this account relatively anonymous so I feel more comfortable being very honest about things. That being said...
The problem was analogous to pricing at Uber or Lyft. The product was very different - not related to transportation in any way. But trying to balance supply and demand through pricing. Supply and demand could vary wildly by the minute and pricing was per customer, so that involved adjusting ~1M prices dozens of times per day. We used control theory, game theory, time series predictions, and lots of other things to understand / control the dynamics of the market and the competition within subsets of the market. I'm convinced the onset of the pandemic would have crushed the company's revenue if this dynamic pricing was not in place. Instead, we merely needed to add some new boundaries to contain what the system was doing (boundaries that were unheard of pre-pandemic like entire sectors of the world economy closing down).
As a software engineer with a BS in Math, I did the first version of this out of intuition without knowing anything about the fields I mentioned above. I was accidentally trying to reinvent control theory without knowing it existed (knowing to do a web search for "control theory" would have helped a lot!).
As time went by and we hired some people with more applicable math knowledge, they were some combination of impressed and amused at what I'd done. And they were able to take it to the next level. Seeing this, and having become the team's manager, I started to hire people with much more advanced math and stats knowledge. Advanced stats people came last and they should have been earlier. It turned out we had shipped multiple experiments that we shouldn't have because we didn't understand that we weren't measuring what we thought we were. As the system got better and better, measuring the impact of our system on indirect market effects became the hardest part of the problem. For example, a change will favor some competitors in the market - is that positive overall? If you think there's a simple-ish answer to that question (like I did at one point), you probably aren't qualified to design / analyze these experiments. wink
Where this eventually led, before I moved on, was the realization that we didn't just need a software architecture, but also a math architecture that helped keep the mathematical dynamics sane and possible to reason about. And it required a lot of effort to help the non-math people in senior management understand this wasn't black magic, but something that needed to be reasoned through by people who understood mathematical rigor (i.e., we needed to continue hiring mathematicians). That's a tough sell given that it's really hard to understand the value of mathematical rigor when you don't have it yourself. We weren't formally proving anything, but when things get complicated, it's really important to understand exactly what assumptions are necessary and when that gets difficult, it's hard to avoid the need for people who know mathematical rigor.