"The Memory Police", by Yoko Ogawa [1]. An island where things get disappeared by the Memory Police - physically from the world and from people's minds. A few people are immune to forgetting, so the Memory Police is on the case.

[1] - https://www.goodreads.com/en/book/show/37004370-the-memory-p...