- Increasing the numbers of servers running the app. App connections were still stacking up though. This gave us more breathing room though for connections to start stacking and handle small spikes.
- The database seemed very overloaded with so many concurrent connections. I began putting everything I could into memcached (we already had a lot of data in it, but I put more).
- now we had a cache hotspot. Some digging found a age old bug in our cache driver where it didn’t actually keep things in process memory after fetching from memcached and we had a medium sized key getting fetch 100s of times per request.
- Days and days of app optimization after profiling. Our average response time went improved by more than 50%. The site would still start collapsing under a little load.
- While profiling in a single request all queries would complete very quickly (<50ms). Somehow the DB was still the bottleneck. We overprovisoned it significantly and it still would collapse.
- I started collecting counts and timings for cumulative and maximum single cache/db, read/writes to our log stack.
- the bottleneck was clearly still the DB.
- at this point we were desperate. Thinking it might have been an issue in the underlying VM we live migrated the DB to a new VM.
- the database was still the bottleneck.
In the end the thing that fixed it? A simple OPTIMIZE TABLE.
Somehow ANALYZE TABLE hadn’t detected anything but rebuilding the table still fixed the issue.
If anyone is looking for a good load testing tool, Vegeta was invaluable. I highly recommend it.
We had this at work, where one customer complained some operation was very slow, taking around 30 minutes. Couldn't pin it down, copied their database to my machine and it took only a couple of minutes. A bit of digging and I found that in this case this module caused a few million of fairly trivial SQL statements to be executed.
Each took less than a millisecond to execute locally, but round-trip time over WiFi can be 10 milliseconds or more. So suddenly 2 minutes becomes over 20 minutes.
I asked how the client connected to the LAN, and it was indeed via WiFi. As a quick fix we got the customer to use a network cable, which did indeed reduce the running time to a few minutes. The proper fix was to a bit of caching.
It can be even worse if development is done using a local database, possibly on super fast local SSD. The latencies can be orders of magnitude lower, hiding performance issues that would be obvious even with only a millisecond of additional latency.
I've seen many vendors claim that they only "support" monolithic single-machine setups (sometimes even virtualization has "unacceptable overhead") when it's blatantly obvious that the application is just written with the assumption that database latency is approximately zero.
Let me ask you this: let's say you're downstream of a very slow server that is outside of your control. You need to access it, and there's nothing you can do to speed it up. Are you IO bound in this case?
I want to say no? If I were to reframe the question as a slow function that I had to call, I wouldn't call it IO bound, so I think not - the fact that there's IO in between the two components is more of a confounding factor in profiling than an actual performance limitation.
It may be that you didn't sleep well last night, or that your stomach is bugging you, but the reason it's going to take an extra hour today is because Tim has a blister.
Whatever is holding up that server is what's holding up the entire train of communication. If that is where $5000 will fix the problem, don't talk about any other problems, you'll just confuse management.
The next place to stop if that doesn't work is, if you can't fix "Tim's foot" can you offload some things "he" is dealing with? Dump some of his load, work steal, etc. But these are just mitigations.
But this is the summary of a large part of that book: Systems have bottlenecks, address the bottleneck. Spending time on improvements elsewhere will not help the bottleneck, you're just wasting time/money. Data/material is going into it too fast to process (work is building up). Improving before the bottleneck just makes its queue fill faster. Improving after the bottleneck just creates a segment of the system that's starved for work. So focus on the problem at hand, once it's addressed, focus on the next bottleneck.
Buffering up does improve things when there is variability the processing time for each task, but it can also make average wait time hell. So you better be sure whether latency or throughput is really your primary concern.
In short I think I/O bound is a concise way of saying that this work will leave my concern and will pass back into it at some later time.
Saying something is CPU bound means that the CPU is the bottleneck. For instance, the data IO doesn’t limit the system throughout of iteratively running a hash algorithm a million times, the CPU does. An IO bound process might be something like reading a lot of data off a disk (or over a network) with a small, quick transformation on the CPU. Where you saturate the data bus, but the CPU is comparatively idle.
If you own both local and remote peer, you then instrument the remote peer. In your hypothetical scenario, CPU saturation would finger the query provided, in which case you are back to correcting that code in the I/O bound node. So, your calling/local node was I/O bound, and your remote/server node was CPU bound.
Sometimes this very same I/O bound pattern -- low cpu utilization on the client node due to blocked threads waiting on I/O from remote peers -- is simply due to insufficient resources and requires changes in capacity planning.
Extending your hypothetical, let's assume as the system in question (post- query optimization) is used over time, the extent in data to be processed grows, with the subsequent result that the optimizied query is now saturating the CPUs while supporting the same number of clients as before. Do you still insist on asserting that processing conditions of a remote peer allow for mis-characterizing the processing condition on the local nodes as "CPU bound"?
The whole system together is CPU bound. But the individual part might be 'IO bound'.
The individual subsystems can be I/O bound on other subsystems whereas considered as an entire system, it could be CPU-bound.
So, yes, one subsystem "blocking" on network I/O definitely is I/O bound regardless of the nature of that I/O, be it database-related or not, it's still I/O.
The system at the other end (the db subsystem) may also be I/O (or CPU) bound, but that is distinct from the clients bottleneck.
Also I'm IO bound, so don't care is a weird response. Doesn't matter were I'm bound, if it needs to go faster, I need to examine what I can do to remove that bottleneck. What I'm bound on just gives me a clue where most of my energy tuning/troubleshooting goes, not shrug and forget the problem.
But maybe I'm just lucky to have always worked with good people.
edit: To be clear, when I say "most people" I mean "many people I've talked to, especially with regards to excuses for not caring about their service performance", and that they are incorrect.