Author here - happy to answer questions and address feedback. I suspect I didn't explain Lisp-in-Lisp as well as I need to, but maybe it's enough to get non-Lispers intrigued. :)
It seems heavier on math than it really is because (I believe) the authors want the reader to see the parallels between math notation (functional, declarative, concise) and Lisp implementation (also functional, declarative, and quite concise).
A person no more needs extensive understanding of mathematics than they need such understanding of accounting or image processing.
That said, domain knowledge is helpful, but so is being able to work in fields which are over one's head. The big ideas are not the maths, but the abstractions which are applicable to many fields.
My advice is focus on the parts you understand [this is Knuth's recommendation for The Art of Computer Programming as well] and plan on coming back later as your knowledge increases. SICP can be digested in small chunks over time.