I've only ever heard "walled garden" used in a negative sense, so I think most people get that it's not meant to be a good thing. Emphasis is placed on the "wall" rather than the "garden".