"Pumping water down hill is more efficient than pumping it sideways"
The CoP of a heat pump is dependent on the temperature potential across the heat pump. Caution to the reader who thinks one could pump heat from frigid external temperatures into their very warm, high temperature house: the very scenario your post suggests they should be used.
Heat pumps are great for increasing thermal potentials and moving heat across them (not for sourcing energy). They're also great for balancing an internal thermal state that is, on-net, in balance (think of a large office building in the morning, one side is being heated by the sun, the other side is frigid -- heat pumps can balance the internal energy demands instead of simultaneously cooling one part of the building and heating the other)