The first problem is that this isn't always obvious. You can't substitute all leafy greens for spinach, there aren't many good non-milk non-fish sources of calcium, not all nuts and seeds have the right amount of vitamin e. It's quite hard to balance everything even when you're tracking individual foods, and it makes for some very strange meals.
The second problem is that a lot of the important nutrients are coming from foods that are otherwise calorie-dense or have other problems. Citrus fruits are sugary, milk and nuts are fatty, uncooked spinach can cause stomach trouble. It's not necessarily bad to eat calorie-dense food, but it does make things less forgiving when you're trying to eat at a calorie deficit to lose weight.
It makes me wonder if maybe I had long-term deficits in some of these nutrients and just didn't know.