I suppose you are right; that's why those truly paranoid filesystem drivers delete the keys from memory after a minute or so.