Yes. Lisp is fundamentally simple and is introduced in the beginning. I can recommend Racket with the sicp language extension to get the dialect used in the book.
How is the math? I've heard it's not exactly easy.
I’m just finishing chapter 1 (1/5th of the book) and the math so far hasn’t been too challenging, but you should certainly go in with the expectation that you’ll do some Googling to figure out some context to complete the exercises.