This is what Google Mail + Calendar are using, including features like RSVP, etc.
It's all there, just passing iCal objects around.
No open-source calendar server that I know of supports this - they all rely on your email client managing this client-side (when you accept an invitation it adds it to your calendar).