I think your missing the subtleties of British English, "can't really blame them" means that you hold them responsible but that you also see their side of things.