Right, if you think about it almost all of the heat from the fire goes right up the chimney. The brick of the firebox & chimney are generally built outside the building envelope for safety reasons, so any heat in the brick doesn't warm the house. The only heat that actually makes its way into the house is the radiant heat cast from the fire itself, which is small.

The smoke is poison, and unlike a furnace, gas fireplace, firebox, or wood stove, there is no heat exchanger, so you can't capture any of the heat out of the flue gasses.

It's counter-intuitive, I know. But this is why everybody congregated around the hearth in the days of fireplaces.