I was brought in to a project on Friday to help figure something out. The gist of the project is that we have a system which generates tons of event data. Basically interaction logging. And until recently the system sent those logs to some other systems via nightly files. But someone in the business decided it made more sense to do it “real time” and so some folks have been diligently coding away and have built their solution only to find there was a gigantic miss:

The system is generating thousands, sometimes tens or hundreds of thousands of events per hour. And the api that is to receive this data can only receive 100 “records” (events) per request, and the system that generates the data can’t do any sort of multi threading or multiple requests simultaneously…it can just fire off one request after another..

I did the math and on our busiest day this process would take something like 827 years to send all of the event data…

I say all of this to highlight your point that it’s a foundational skill to gather requirements..and yet I’m working with sr engineers that created this mess…