I beg to differ mostly. The criminal justice system is heavily staffed by people of all colors and ethnicities, including many, many blacks, who in some cities predominate among the police (and general population, at least in some neighborhoods). Despite this, it's often just as bad towards civilians and minority civilians as a mostly-white police force.
More specifically, the criminal justice system is classist, and that minorities are often part of the poor and underclass in many cities makes them much more targets than their coincidental skin color, though it sometimes seems to serve as a useful visual marker for police to who it's easier to target on sight. The idea of so many police officers and other law enforcement officials who are themselves black or some other visibly non-white ethnic group nonethless targeting civilians who are of the same color, for race reasons, doesn't really make sense from a racism perspective, but it does make sense from a class perspective.
Racism and classism feed each other. We've known that since even before the civil war. Claiming classism doesn't make racism - poof - disappear. It actually reinforces it.
Spreading the net a bit wider, you can also look at the recent and massive ICE crackdowns on illegal migrants (and sometimes US citizens along the way). Just by looking at photos of these incidents, you quickly note that many ICE agents are themselves black, Latino, Asian, etc, enforcing draconian crackdowns against other visible minorities. That's not simply something you can label under racism and be done with defining it. Other systemic factors are at work there.
That's because there's two types of racism: individual, and systemic or institutional.
Individual racism is the low-brow obvious type of stuff. Slurs, clutching your handbag walking next to a black person, that type of thing.
Institutional or systemic racism is more abstract, but also much more harmful as a whole.
Take, for example, the DEA. Marijuana is a schedule 1 drug despite not being nearly as harmful as even most schedule 3 drugs. That's not a coincidence.
That reflects the widespread institutional racism of the DEA. Marijuana was chosen to be scheduled 1 because of its association with black Americans, deliberating inflicting more widespread harm onto them.
Okay, so why does this matter? Because within racist institutions, you yourself are forced to be racist. Even just existing in the institution is an act of racism, similarly to how working for a military contractor is itself an act of support of War.
These institutions have a culture and set of expectations and rules, and to exist within them you must comply.
For example, you cannot be a police officer and simply choose not to criminal marijuana and instead criminal white drugs like cocaine. You have rules, and you must follow them.
You being black does not override that. You being Asian doesn't override that.
So, the big picture. ICE, as an institution, is racist and has goals to particularly harm specific racial minorites. It mobilizes on these goals via its policy, it's expectations, and even it's culture.
Being a brown ICE worker does not detect from those goals, and just by existing in ICE and doing their bidding you are implicitly racist. Because the institution is racist, and you support it. And their goals are racist, and you're a big part of making their goals a reality.
As a side note, this is also why "I have a black friend" arguments don't work. That's a refutation of individual racism, not institutional racism.
For more examples of systemic racism throughout US history, please see: redlining, gerrymandering, Jim Crow, segregation, the FBI, and the CIA