This is a serious impediment to developing any useful expertise in any area, but I have sort of made peace with this tendency. But not all is lost. It has been my experience that the next time I visit a previously learned topic, it's significantly easy for me to load everything to my "active" memory, and build-up from there. Apparently, you never truly forget the things that have once learned well. See here, for e.g.: https://www.sciencedaily.com/releases/2008/11/081117110834.h...
> Scientists at the Max Planck Institute of Neurobiology have been able to show that new cell contacts established during a learning process stay put, even when they are no longer required. The reactivation of this temporarily inactivated "stock of contacts" enables a faster learning of things forgotten.
I am no expert on memory, but from what I have read on the topic, our long-term memory is virtually unlimited, the main constraint being inability to recall rather than the memory content being erased. If you have once learned a topic well but haven't visited in a long time, you may not be able to recall it very easily, but you will be able to recognize or relearn it better than someone who doesn't have any exposure to the topic.