One of the most "mind boggling" areas of math is number theory

Math dealing with real/complex numbers has been vastly explored, and there are very powerful tools, and computers are very good at these kind of problems (Calculus, etc)

With number theory, progress is much harder, things work in a completely different logic (think for example that 5+3 can be 0 for example)

One of the places where these mysterious sides of math comes together is this: http://en.wikipedia.org/wiki/Riemann_zeta_function