The task at hand wasn’t easy at all, it required both:
1. Making numerous network requests (some endpoints support batching, so do that as much as possible) 2. The entire main loop needs to be as fast as reasonably possible
The only real way to satisfy both, given that serially executing the network activity would waste tons of time, was some fairly tricky concurrent logic. I ended up using a few queue structures to collect day before making batch network calls, and stitching together all of the results at the end (block on all work threads, collect, report, then start next iteration)
Every step along the way made sense, every addition of complexity felt justified. And the first person to come to me asking how the fuck it got so complicated is frustrated, so my knee jerk reaction is to get frustrated back saying well how else could we satisfy complex requirements with a complex solution.
We should always be challenging ourselves and others to keep solutions simple, but remember that the simplest solution may itself still be complex.