The fridge is simply moving a little of the warmth in your room out of its box, and adding a little more warmth to the room in the process. You lose no heat from the room, which is what matters, really. It would be worse to dump the heat outside! At least in winter. The heat has to be removed in summer if you have air conditioning.
In any case, an exterior heat exchanger and heat pump that can handle a wide exterior temperature is much more heavy duty. This could all end up being less efficient, in practice. House-scale heat pumps can efficiently move heat even from a freezing cold outside into a warm interior. Just as fridges can efficiently move heat from their cold interiors to a warm room.