It is worth explaining just what is wrong with it. Mercury of course is a toxic compound, but kids used to crack mercury thermometers in their mouths and suffer no consequences. Pressure gauges came with exposed tubs of mercury and you would get some on your hands and it was fine, even kind of fun to play around with. Felters famously felt symptoms after careers of using it, usually blamed on long term inhalation, but it required a long time and lots of exposure to get to that point. Many people received dental amalgams with mercury bases and while there has always been a conspiracy theory about it, it has never been acutely toxic enough to be mainstream—the shift to other composites has instead been driven by comfort (metal fillings conduct heat and cold at an unnatural rate) and aesthetics (the new composites are white instead of silvery).
The basic reason is that the body does not eat metal and does not know what to do with it largely. Even with the metals that the body uses, like cobalt, it wraps every atom of cobalt in a big old ring of carbon and that's Vitamin B12.
Dimethyl mercury, is different from normal mercury, in a similar way. The methyl groups are organic groups that your body is much more willing to absorb and process. So the latex is not protecting you nearly as much because latex is this organic natural filter that lets things through... But also your skin is not protecting you because this stuff can seep into your skin via those methyl groups, your cells are just like “oh you look interesting I can take you into my chemistry.”
So yeah, milligrams are all it takes, tiny trace amounts: but the reason that you don't think of it being quite that dangerous is that your exposure to the liquid metal version is an exposure to a form that your body is uniquely unsuited to process or absorb. Water off a duck's back. Even though the water contains neurotoxin, it just rolls right off. Even, in our case, when ingested. But add these little methyl groups, methylmercury and dimethylmercury and really any organic mercury molecule, it gets real bad real fast because it's no longer rolling off our backs, it's soaking into our feathers.