The relevant word here is 'stay'. Obviously a loaded fridge will take longer to reach the required temperature, but once the correct operating temperature is reached the only work the fridge needs to do is remove any heat that percolates through to the inside of the fridge from the outside of the fridge. That is solely dependent on the heat-conductivity of the fridge walls, and has nothing to do with whatever is inside the fridge itself.