See, I'm all for big data and part of me loves the idea of analyzing these frames, but I wouldn't want to do it for my life. If you could constantly go back and re-evaluate every moment of the past, I think it would drive you crazy. Hell, my friend spent 4 months organizing 1 month of photos from a holiday. If you had your whole life to work with, forever, you could just get lost in the past and never come back.
Maybe an hour is too short, but I'd definitely want things I hadn't explicitly saved to 'decay' in some way; either a rolling window, or deleting one in n frames based on some measure of interestingness - where n is a function of time. It's not just a matter of hiding the old ones, I think it would be vital to know that deleted images were really gone - to give some urgency to actually looking at them, and to reduce potential time wastage looking at thousands of unique - interesting to a machine - but unmemorable photos.