There are a few problems: People don't know or care about profiling the basic stuff like context switches and data copies; people don't have a baseline idea of what should be possible; abstractions upon abstractions upon abstractions even when it complicates the code.
One of my biggest pet peeves in that respect is that people tend to not even know how many objects they create in many dynamic languages. Many people don't even know of any simple ways of finding out.
My most successful application speedup (EDIT: measured in speedup per hour expended - plenty of examples of much greater speedups, but they are rarely as quick) was spending an afternoon eliminating string copies in a late 90's CMS written in C++, and cutting page generation time by 30% in a system that was already heavily optimized (it provided a statically typed C++ like scripting language that was bytecode compiled at a time when most competing solutions were using horribly inefficient interpreters, and were often themselves written in slow intepreted languages).
My second biggest pet peeve is when people think system calls are cheap because they look like function calls. Sometimes you can increase throughput by an order of magnitude just by eliminating small read()'s (non-blocking filling of userspace buffer instead, and "read" from that). When I see slow throughput, the first thing I do is break out "strace" to look for unnecessary system calls...
But these problems are often a direct result of abstractions that means huge numbers of developers don't know or understand the cost of a lot of the functionality they depend on any more, and can get away with that "too often" because they work on hardware that is so fast it is extremely forgiving (e.g. my new "weird unknown Chinese brand" 8-core phone does about 10 times as many instructions per second as my ISPs total computing capacity in 1995, and it's nowhere near the peak performance of todays flagship phones), ignoring the fact that at "web scale", things are not as forgiving any more: server costs quickly start adding up.
But of course, most of us most of the time work on stuff that needs to handle little enough traffic that paying an extra $20k for hardware is cheaper than spending the developer time to optimize this stuff. Ironically this makes the optimization more expensive: Fewer developers ever get to obtain the experience to optimize this stuff at scale.