The brick works as a thermal battery - if it's cold, you're heating your house, and the intermittent exhaust is keeping the brick warm. If you're not heating, well it's not cold so the brick temperature doesn't matter much.
But: we need to build low energy housing to manage climate change. That means there is very little waste energy available. Last house we built was 230m2, family of 5, had a total energy consumption, including everything from heating to water to home electricity, sitting at 8000kWh/y. That's in southern Sweden (N58deg).
There is no relevant amount of waste energy available.
This is the kind of housing we'll need to meet future energy targets.