I do have exactly the same problem/need and would add the following requirements:

1/ add history per contact: e.g. met X during event Y and talked about Z.

2/ integrate w/ my prefered mail application and other communication means (twitter/fb & co) to keep track of the interactions I had w/ the person in question.

Anybody interested in building that? I'd be more than happy to help :-)