We had to do this recently as well. You have to get rid of him. A poor performer in your staff will drag down morale and destroy productivity. Keeping him on will send the message that you don't care about performance.

It is acceptable for you to fire an employee that is not performing at the expected level, in the same way it is acceptable for a new employee to leave after a short period of time if the job doesn't feel like a good fit.

Removing him will assure everyone else at the startup that you care about the quality of people you keep around; it's very hard to be dedicated if you know there are poor performers dragging things down.