It's not just dirt. Traditionally people lived in close proximity to a variety of domestic animals. Most viruses originated in animals and then crossed to humans. So our immune systems evolved to cope with that exposure.
And IIRC, the hygiene hypothesis mostly has to do with autoimmune diseases and our immune systems not learning how to differentiate between serious threats and harmless exposure.