Not sure without knowing more of your background - provisional answer is "maybe." Some parts of Ch.1 are certainly math-heavy. The video lectures are quite accessible. Did you study, say, ~1 year of calculus?
It seems heavier on math than it really is because (I believe) the authors want the reader to see the parallels between math notation (functional, declarative, concise) and Lisp implementation (also functional, declarative, and quite concise).
A person no more needs extensive understanding of mathematics than they need such understanding of accounting or image processing.