The cortex is never "empty". The blank slate model is an outdated speculation with no empirical support, akin to "you only use 10% of your brain" or "you can tell things about people's personality by feeling the bumps on their head".
[0] A. Roe, S. Pallas, Y. Kwon, and M. Sur, “Visual projections routed to the auditory pathway in ferrets: receptive fields of visual neurons in primary auditory cortex,” J. Neurosci., vol. 12, no. 9, pp. 3651–3664, Sep. 1992.
PS: I'm not saying that there is uniform initial connectivity. Just, the prediction level units (cortical columns) are unable to predict anything until they learn "something".