This is why the technology is best combined with geothermal concepts. You only have to go a few feet deep to get warmth and that solves basically every problem a heat pump has.
Using the ground as a heat source requires a relatively large installation as you need to make it big enough to draw heat from a large area, even as you use it year after year.
However, you can also use the ground as heat storage, and pump back heat during the summer. This greatly reduces the size, as you are not at risk of "exhausting" it.