So it very well could be that thinking on a worn neuron, trying to do memory access on it, might damage it even more (leg analogy)? It could be possible that the "swelling" could be redirecting brain to use other paths but with caffeine you're utilizing a worn-out neuron.

Maybe with chronic deprivation + caffeine habit, you're weakening your cognitive function - repeatedly using a misfiring neuron. At least, with no caffeine, you'd feel your memory is hazy and hard to recall, go to sleep.