Alas, a walled garden is not the opposite of freedom. Indeed, in many places, living in a walled garden will make you factually freeer than your fellows outside.
You're constructing a scenario to justify an abstract argument. The wall is the only salient feature in the walled garden metaphor. There's no terror lurking outside.
If you mean to imply that Apple's oversight has some advantages, such as security, that's a separate issue.
In the context of Apple's app store rules, the wall refers to Apple's rules. The absence of those rules does not reduce your freedom as a developer; you're can still abide by them without Apple requiring your to do so. But you have more choice without the rules.
A game is only a game if all the players have a shared set of rules and can trust each other to stay within them. As soon as someone breaks the rules the game is over, and the freedom provided is lost.
For example, justice in a free society should not be arbitrary and capricious. It should be fair and equal, and applied with discretion. A society where minor rules are difficult to understand but are enforced to the maximum extent is a tyranny.