You'd think so, but I've taught ESL math. Good luck getting them to understand what's even going on without speaking their language. Especially if some, like my students, come in at high school having only really had five years of formal schooling.
You need the language to teach the math, basically. Also, not all of the kids I had were literate in their first language (A Mayan language, so most weren't) or even their second (often another Mayan language) or third (Spanish usually), so you can't just say 'Give them a workbook'