From the toxicity section (https://en.wikipedia.org/wiki/Borax#Toxicity) it seems like the concern is most about ingesting borax in large quantities or breathing in the dust. And also being cautious on potential health affects:

  Since people are already exposed to boric acid naturally through their diets and water, Health Canada advised that exposure from other sources should be reduced as much as possible, especially for children and pregnant women. The concern is not with any one product, but rather multiple exposures from a variety of sources.