The brain is highly volatile both chemically and physically and is changing minute by minute. The notion that a memory belongs to a single group or cluster of neurons is absurd. They are constantly being reordered and reprioritised and changed throughout the lifetime of the animal. And being lost due to chemical damage and the effects of ageing and free radicals.
The nature of memory and conciousness isn't just the physical existence of neurons. Who you are is more than the sum of the parts.
The mind is also as much a process of the body -- your conciousness and mind grows within it from birth and a large part of who you are is as much a product of that, its hormones and its metabolic chemistry as anything that resides solely between the ears alone.