How can you turn it off without turning off history ("My Activity") altogether?

I noticed the "memory" too and it's turned Gemini into a useless syncophant for me, but so subtle that I almost didn't spot it.