If your IPMI is in any way exposed to anyone other than your administrators, then you have other problems. These interfaces should be segmented away from all other networks, irrespective of any vulnerabilities they could have.
But I agree that you should filter all the relevant ports in all networks, just in case somebody screws up.