Anyone who ignores structural racism and espouses the policy positions that Murray does is either ignorant or racist. Murray, being a public figure whose ideas have had real impact on the material conditions of marginalized people, does not have the excuse of ignorance.
I'm speaking from personal experience here. I used to be on the right-libertarian train myself before I realized that oppression was primarily structural and that a laissez-faire approach always perpetuates it. I genuinely do not believe one can be non-racist and espouse Murray's policy positions once one understands this basic fact.
If you want to learn more about how welfare reform in the 1990s—something Murray influenced a great deal—affected millions of lives, I'd recommend the first season of a podcast called The Uncertain Hour.