I agree.
I don't necessarily buy the "loneliness" argument for cognitive decline.
However I certainly buy the importance of keeping mentally and physically active.
Infact I would go as far as saying people who use the "loneliness" term are actually just using it as a lazy synonym for "lack of mental and physical activity". You don't necessarily need to surround yourself with people to achieve it.
Whether its still working in a job, or doing non-job activities such as gardening, travelling, photography, I think keeping your brain busy and your body moving is the key.
If you revert to the cliché retirement image of taking long naps during the day, stopping only to eat and watch daytime TV, then IMHO that's where the problems start.