I am sure it already knows a lot regardless of the memory feature, as long you're sharing your chat history/ have your history enabled, but I agree, it'd simply worsen it.