Tricky question.

At the moment destroying these means destroying people's external memory. The "promise" was to keep them for us, so we needn't have to worry of keeping it ourselves, and so we can "live" more, since there is less to worry about.

Many will only realize years later how bad this is, and will try to re-live an experience which they physically attended to, but not really took apart in it - just to find that all of this is gone.

So at one point, it's a good thing that they are gone: people will learn not to rely on them. From another perspective, it will result it a lot of lost memories.

But hey, we need to learn our lessons somehow.