If you graphed the visit rate to our shopping cart page it would follow a curve that was sort of like a sine wave (well, a sine wave plus a constant because you can't have negative cart traffic) with a 24 hour period. I'm just going to call it a sine wave.
The amplitude varied seasonally, and was different on weekends and weekdays, but generally any given day was a sine wave.
If you graphed order rate it also followed a sine curve too, in phase with the visit curve and generally with a fixed ratio between them. So if the cart was getting N visits per minute, we'd be getting fN order per minute for some number 0 < f < 1, and f was pretty much constant.
This continued as the business grew, so the overall amplitude of the visit curve increased. But at some point the order rate seemed to hit a wall.
What we would then see is that as the visit rate was heading toward its peak for the day when it got to a particular fraction of that peak, say 80% of the peak, the order rate would stop tracking the visit rate and would become flat. It would stay flat as the visits peaked and declined until the visit rate was back down to 80% of its peak and then orders would resume tracking visits.
So far nothing too weird about this. What was weird is that when we'd have a day of especially high cart traffic such as when something in the news caused a big increase in demand for a few days for the kinds of things we sold, and we'd get 3 or 4 (or sometimes even 20 or 30) times the normal cart visit rate throughout the day--the plateau on the order rate would still start when the visit rate hit 80% of the visit rate peak for that day and go away when the visit rate fell back below 80% of peak.
The order processing was handled by a different server than the shopping cart. They did use the same database for product and customer information, but we could not find any signs that the database was a bottleneck.
This just made no sense. Suppose we'd been seeing on normal days that visits would peak at 1000 per minute, and orders were 10% of visits. Then we'd see visits grow from 0 to 800 and orders grow from 0 to 80. Then orders plateaued at 80 while visits continued on to 1000 and back to 800. Then as orders fell from 800 to 0, orders fell in step from 80 to 0.
Now suppose the next day we had 10 times the activity due to some news item. So the next day visits rise from 0 to 10000. The order system would now not plateau until the visit rate got to 8000 and the order rate got to 800.
And then when the increased demand was over in a few days and we were back to 1000 visits peak the order system would again plateau at 80.
So whatever was causing the plateau was somehow tied to the visit rate, making the plateau depend on what the peak visits would be for that day.
But how on a given day when the visits would first rise to 800 and the orders to 80...how did it "know" if the day was going to be a 1000 visit peak day or a 10000 visit day so it could "know" if it was time to plateau?
We never figured this out. Reasoning about our systems and their dependencies I could not figure out any way in theory that this could be happening nor could I find anything no matter how much logging and tracking I added all over the place.
Everything I could think of that could even remotely cause a plateau would have caused a fixed plateau, not some weird plateau that's level gets set daily based on what the future peak visit rate is going to be for that day.