This is really in poor taste. Can you imagine if the email to Bob said "hey lets meet at the pub at 6" and the email to Alice said "hey lets meet at the pub at 4".
I know this is unlikely, but sending an email that looks like each party is getting the same thing but in fact are getting different emails just is bad.
The tracking thing is not a big deal, seems like everyone does it. Just block images and you are golden.