I wish I understood this, but lack a lot of the background in CPU architecture beyond a very superficial understanding (ex. what is meant by "branch prediction"). Can anyone recommend me a good source to help me get up to speed on this?
Conditional branches are taken and go somewhere else or they are not taken and just go to the next instruction.
Branch prediction (really conditional branch prediction which is a form of speculative execution, think Spectre) is making that guess. Getting it right makes the code fast. Getting it wrong costs about 10-15 clock cycles and some lost energy. You can and Intel does throw anything at it, neural networks based on the stack return addresses, recent branch history, … anything.
https://www.lighterra.com/papers/modernmicroprocessors/
A bit dated, but the major ideas used in current CPUs are all covered!