They loose heat to the outdoors as you say and since they cool down a lot faster than the internal chimneys it is also generally harder to get the fire going when re-lighting. The chimney on my house is on an outer wall and when it's below 0C I have to light a fire every day or it becomes too much of a hassle to get it going.
Houses in southern states were built with external chimneys which made sense for a couple reasons. Back in the day people would keep a fire 24/7 for cooking and that fire + southern heat is uncomfortable. Because of the shorter cold season the chimney was on the outside of the house. Also, in the case of chimney fires you could tie a chain around an external chimney and rip it down with a horse, hopefully saving your home.
Last place I rented with a fireplace, there was a metal door in brickwork below it in the basement. It wasn’t until after I moved out that I realized what it was for.
A massive stone fireplace that's heated with a single load of fuel, then has its chimney flaps closed to keep the heat from escaping, is going to be much more efficient. I wonder if maybe the fireplaces in many parts of the USA are actually designed to not heat the house too much, given the usual weather in e.g. Texas.