I see one way you might be able to work around this: if you made a suit out of this stuff, and were somehow able to rig it so that all of the heat is dumped inside the suit, then it would be "invisible" to IR sensors. However, it would get really uncomfortable inside that suit really fast.
Also, even if it somehow magically absorbed all of the IR without re-radiating, it would still be possible to build IR sensors that detect it: if it's perfectly black in IR, it will show up as a "shadow" against a warmer background. To be invisible to IR, you need a suit that matches its temperature to the surroundings.