Responsibility is really a kind of duty. When we say that you are responsible for your own behavior, we are not saying that “you are the sole cause for your own behavior”, we are saying that you have a duty to control your behavior. Duty is a moral concept, not a factual concept. So if you think that this assumption at the very core of criminal justice is “false”, you have misinterpreted it as some kind of statement of fact (which would be refutable).
It is not a statement of fact, you can’t refute it.