Actually, in this example knowledge of the world (I'm interpreting this as referring to knowing the details of how "files" and "hard drives" relate to each other) is not necessary. The ambiguity disappears as soon as you know the meaning of "fit" -- fitting requires a large thing to contain a small thing. Therefore if something is too large, it must be the contained object, and if something is too small, it must be the container, and those roles are marked directly within the syntax of the sentence.
You can easily see this experimentally by asking people about sentences with nonsense words:
1. The glirp couldn't fit on the vell because it was too small.
2. The glirp couldn't fit on the vell because it was too big.
Then again, I see you've already noted that speakers who don't know what files or hard drives are should have no trouble with these sentences. Is the lexical meaning of "fit" "knowledge about the world" to you?