As soon as you fetch the cache line, every other core will know you have done that via the cache coherency protocol. You have just disclosed an address potentially derived from some sensitive information.
Edit: to be precise they can, for example, see the transition from exclusive to shared.