If you’re on Mac/iOS, contact journal may be exactly what you’re looking for — it adds tasks/reminder/attachments/etc to contacts.