I can't think of any good books of the top of my head - I've picked this up largely from starting out with assembly, and spending years picking apart compiler output (initially because the C compilers I had access to were junk, and I more than once would compile something, disassemble it and then hand-optimise the result...)
One approach is to learn to read assembly (being able to write it from scratch is optional), and understand what affects clock cycle count. On modern architectures that also means understanding cache effects, and particularly looking out for memory access patterns. Then look at lots of disassembly and learn to understand what kind of code typically gets generated from your C. Often it will be plain obvious why things are slow when you see the amount of generated code (though here be dragons - sometimes modern compilers will actually do a good job optimising, and as a result produce code that no sane human would ever write manually).
You can quickly get a good feel for patterns that are good and bad.
But in general:
- Profile. Profile some more, to make sure you're actually spending your time on the code that matters.
- A useful rule of thumb is that for any given algorithm, the fewer function calls and memory accesses, the better it is likely to to be. Functions calls are expensive, and often (not always) reduce locality of code, and so increase the amount of time spent accessing memory that is not in cache (== expensive). But don't start hand-unrolling/removing abstractions until you know by profiling that the code in question is performance critical, or you may just end up creating a maintenance nightmare.
- System calls are particularly nasty on any memory protected modern OS. The context switch overhead tends to be absolutely brutal. Which means it's very often worth quickly breaking out strace (on Linux) or equivalents to quickly eyeball the system call behaviour of applications before spending much time on other stuff. E.g. one particular pet peeve of mine is the amount of time you see read() or write() with small byte counts - if that occurs often, you can sometimes get truly massive speedups simply by changing the code to do non-blocking read/writes to/from a buffer and do smaller read/write's in user space from those buffers. (Always be suspicious of code that does read/write/recv/send or similar unless you can see it uses large-ish buffers... but profile). My experience is that odds are shockingly high that people will severely underestimate the cost of system calls.
- Keep data that is accessed together close in memory. E.g. cluster values by access patterns when you can, to avoid thrashing cache when you access the data. This can be tricky, and the solutions can be non-obvious because the actual access patterns may not be that easy to spot.