Ask HN: Learning the math behind cryptography

I've taken an interest in cryptography and have been reading quite a lot on the subject lately. However, my understanding of the mathematics involved needs a little bit of work. I've always excelled in the required math for school up through advanced calculus, but a lot of this is completely foreign to me. Can anyone suggest a good source for teaching one's self the math behind cryptography theory? I've looked up unfamiliar terms and made sense of everything I've read, but I still don't feel comfortable with the field in general. Specifically, Galois fields, relative primality, discrete logarithms, quadratic residues, generators, etc...

If anyone knows of good places to learn about these kinds of topics, they'll be greatly appreciated. I generally find purely academic math theory hard to read as it is, even when I understand all the underlying principles.