Actually, yea, I think the river is quite clean. Especially now that I've seen bodies of water that have an oily sheen on them - and ones that sting if you get them on your skin. I think I'd probably be pretty adverse to swimming in the Charles - but I'd definitely prefer it any day over False Creek in Vancouver.
My biggest concern with questionable water these days is always the bed they rest on - usually the water tends to be pretty clear even in heavily polluted areas (though oil floats so some forms of pollution are quite contrary to that) - but all the really dangerous stuff tends to slowly separate out and settle on the bottom. It's when kids get in and kick up the muck at the bottom of one of these rivers that I get most concerned.
Also yea - this is pretty much all anecdotal as well.