That is where the loading bays are, so they keep the fridges nearby so cold items have less chance to heat up. Plus milk is often stocked from behind, which can only be done where they have the extra space at the back of the store.
The milk is still in the back, along with perishable juices. The difference between them and the aisles is that processed foods (frozen dinners, vegetables, snacks, brand name yogurt & cheese, ice cream, breakfasts) are all located near the front of the store, while fresh ones (milk, meat, fish) are located at the back, and perishable-but-non-refrigerated goods (fresh fruits & vegetables) are off to the back & side. So it really seems like a deliberate attempt to put the items you would buy frequently as far away from the door as possible, and make you walk through the goods that you might stock up on on impulse.
I would bet on consumer psychology over logistics here.
Logistics: because then you don't need rear access to the cooler shelves. That's square footage that you can better use in other ways. (In school I worked in a supermarket that put dairy in the beer/soda cooler, and we stocked it from the back.)
Milk is heavy. It's stocked onto shelves slightly angled forward so the remaining stock slides to the front for ease of reach. This need isn't generally applicable, or desireable, for other refrigerated/frozen foods.
Although, maybe that's an argument for putting it by the checkout lines instead, which I suppose are by the entrance. Hm.
And are you sure milk needs to be stocked from behind? Is it at the back of the store because it needs to be stocked from behind - or is it stocked from behind simply because its at the back of the store?
Plus with thing like milk, it is a real pain to push several gallons uphill to put something else in front (ever try to put a milk container back on the sloped shelf? Takes a couple hands and some fidgeting to do it).
The distance from bay to fridge plus fridge to the exit is the same wherever it goes.