Potential output has nothing to do with the question whether the training data is "memorized" in the weights or not. To me, the answer to the latter question is "obviously yes".