Yes, for dirt I wouldn't worry at all... but e.g. rodent excrement is quite dangerous: leptospirosis (kidney damage, meningitis, liver failure), hantavirus, salmonella, LCMV. I know, chances are low but I'd be happy to reduce them as much as possible.