The standard human digestion system is really really good at breaking down sugar very quickly. Enzymes dominate the process, not microorganisms.
http://www.ncbi.nlm.nih.gov/pubmed/17927751 http://www.ncbi.nlm.nih.gov/pubmed/11157352
If the lactose passed through intact, it wouldn't be a problem. If enzymes weren't responsible for breaking down sucrose and there were bacteria doing it, HFCS would probably be even more popular.