I actually don't think you should ask him to leave yet.
Because what it sounds like is that you haven't communicated very well with him (understandable - you've been busy).
He may not even know that you guys are having issues with his performance. I'd sit down and talk to him first before deciding anything.